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PREFACE 


This  technical  information  memorandum  presents  the  evaluation  procedures,  test 
results,  conclusions,  and  recommendations  for  ground  and  flight  tests  from  the  Have 
SYCLOPS  test  project.  Testing  was  conducted  by  the  United  States  Air  Force  Test  Pilot 
School  (USAF  TPS)  Have  SYCLOPS  Test  Team  at  the  Air  Force  Flight  Test  Center, 
Edwards  Air  Force  Base,  California.  The  USAF  Test  Pilot  School  and  Air  Force 
Research  Laboratory,  Human  Effectiveness  Directorate  (AFRL/HECV)  sponsored  this 
project. 

The  authors  thank  Lieutenant  Colonel  James  L.  Wertz,  the  Have  SYCLOPS  Test 
Pilot  School  staff  monitor,  and  Veridian  Flight  Research,  especially  Mr.  Tom  Landers, 
for  their  outstanding  contributions  to  this  effort. 


EXECUTIVE  SUMMARY 


This  Technical  Information  Memorandum  (TIM)  presents  evaluation  procedures, 
test  results,  conclusions,  and  recommendations  for  ground  and  flight  tests  of  the  Have 
SYCLOPS  project.  The  objective  was  to  determine  if  the  addition  of  the  Advanced  Non- 
Distributed  Flight  Reference  (ANDFR)  symbology,  containing  own-ship  status 
information,  to  a  helmet-mounted  display  (HMD)  aided  in  operationally  representative, 
off-boresight,  air-to-air  and  air-to-ground  piloting  tasks.  The  Responsible  Test 
Organization  (RTO)  was  the  412  TW.  The  test  was  executed  by  members  of  Class  02B 
of  the  USAF  Test  Pilot  School,  operating  as  the  Have  SYCLOPS  test  team. 

Tests  were  conducted  at  the  USAF  Flight  Test  Center  (AFFTC),  Edwards  AFB, 
California.  Ground  tests  were  executed  from  9  to  14  April  2003  and  flight  tests  occurred 
from  10-23  April  2003  on  the  Air  Force’s  NF-16D  Variable-stability  In-flight  Simulator 
Test  Aircraft  (VISTA).  Veridian  Engineering  Flight  Research  Group  and  the  Have 
SYCLOPS  test  team  accomplished  ground  tests,  calibration  sorties,  and  flight  tests  at 
Edwards  AFB.  Two  calibration  flights  and  twelve  VISTA  flight  test  sorties,  totaling  19.3 
flight  hours,  were  flown  in  support  of  this  project.  Ground  testing  and  two  calibration 
flights  were  accomplished  prior  to  flight  test.  This  project  was  part  of  the  curriculum  for 
USAF  Test  Pilot  School.  Air  Force  Research  Laboratory,  Human  Effectiveness 
Directorate  (AFRL/HECV),  requested  this  testing. 

The  ground  and  flight  tests  evaluated  the  modifications  made  to  a  previously 
tested  flight  reference  symbology  set  intended  for  use  in  fixed  wing  aircraft  HMD.  The 
HMD  provided  visual  information  to  a  pilot  through  an  image  projected  onto  the  pilot’s 
helmet  visor.  The  projected  information  could  be  viewed  regardless  of  the  pilot’s  head 
orientation.  The  ANDFR  symbology  was  designed  to  provide  a  continuous,  composite, 
ownship  state  “information  stamp,”  including  airspeed,  altitude,  attitude,  and  heading 
regardless  of  pilot  line-of-sight  (LOS)  or  head  movements  during  tactical  engagements. 
This  symbology  was  developed  by  AFRL  to  address  and  reduce  clutter  and  occlusion 
problems  within  the  head  up  display  (HUD)/HMD  and  to  reduce  the  scanning  time 
associated  with  pilots  visually  collecting  ownship  status  information. 

This  project  used  the  HMD  on  VISTA  to  examine  two  different  configurations  of 
the  ANDFR  symbology  during  both  ground  and  flight  tests  and  compared  them  to  the 
“Baseline”  (BL)  off-boresight  HMD  symbology  set  currently  proposed  for  the  Joint 
Strike  Fighter  (JSF).  The  MIL-STD-1787C  HUD  symbology  set  served  as  the  on- 
boresight  baseline  for  comparison.  The  BL  symbology  set  consisted  of  the  distributed 
airspeed  and  altitude  indicators  from  the  HUD.  The  second  configuration  was  the 
Baseline  Plus  (BL+)  which  added  the  Arc  Segmented  Attitude  Reference  (ASAR)  and  the 
HUD  heading  tape  to  the  BL  set  in  the  HMD.  The  third  was  the  complete  ANDFR 
symbology  set  as  designed  from  the  Have  ATTITUDE  test  team’s  flight  test 
recommendations  in  2001. 

All  test  objectives  were  met.  Overall,  the  ANDFR  symbology  was  found  to  be 
less  beneficial  to  pilots  than  the  BL  and  BL+  symbologies  during  off-boresight  air-to-air 
and  air-to-ground  operationally  representative  tasks.  The  test  team  recommended 
additional  development  and  testing  of  the  BL  and  BL+  symbologies  during  other  than 
day  visual  meteorological  conditions.  Suggestions  for  improvement  and  enhancement  of 
BL  and  BL+  symbologies  were  developed  and  presented. 
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INTRODUCTION 


GENERAL 

A  Non-Distributed  Flight  Reference  (NDFR)  symbology  set,  intended  for  fixed-wing  aircraft 
helmet-mounted  display  (HMD),  was  evaluated  by  the  Have  ATTITUDE  test  team  in  March  and  April 
2001  (Reference  1).  Revisions  were  made  to  this  symbology  based  on  Have  ATTITUDE’S 
recommendations.  Two  variations  of  this  revised  symbology  were  evaluated  and  compared  to  the  MTT .- 
STD-1787C  HUD  symbology  set  using  the  NF-16D  Variable-stability  In-flight  Simulator  Test  Aircraft 
(VISTA).  Testing  was  performed  at  the  USAF  Flight  Test  Center  at  Edwards  AFB,  California,  by  the 
Have  SYCLOPS  test  team  from  the  USAF  Test  Pilot  School  (USAF  TPS)  in  March  and  April  2003. 

USAF  TPS  and  Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate  (AFRL/HECV) 
sponsored  this  test  program  as  part  of  the  TPS  curriculum.  All  testing  was  supported  under  job  order 
number  (JON)  A03HR000.  Six  VISTA  ground  simulation  familiarization  and  test  sessions  were 
accomplished.  Two  VISTA  calibration  sorties  and  twelve  VISTA  test  sorties,  totaling  19.3  flight  hours 
were  flown;  as  well  as  three  target  sorties,  totaling  3.5  hours. 

This  project  was  conducted  under  the  authority  of  the  Commandant,  USAF  TPS. 

BACKGROUND 

The  proposed  flight  test  evaluated  the  modifications  made  to  a  previously  tested  flight  reference 
symbology  intended  for  use  in  fixed-wing  aircraft  HMD.  The  revised  symbology  was  designed  to 
provide  continuous  own-ship  status  information  with  more  precision  and  trend  information,  including: 
airspeed,  altitude,  attitude,  and  heading  regardless  of  pilot  line-of-sight  (LOS)  or  head  movements  during 
tactical  engagements. 

Most  symbology  sets  distributed  attitude,  airspeed,  altitude,  and  heading  information  across  the 
HMD  field-of-view  (FOV)  (Reference  2).  Some  symbologies  provided  the  information  in  close 
proximity  to  an  attitude  reference,  but  the  digital  information  was  presented  on  the  outside  of  the  attitude 
reference  as  in  a  MIL-STD-1787C  HUD  (Reference  3).  Distributing  flight  information  across  the  display 
FOV  posed  two  potential  usability  disadvantages.  First,  aircraft  state  information  distributed  across  the 
helmet  FOV  posed  a  threat  to  pilot  usability  due  to  the  information  becoming  occluded  by  tactical 
symbology  which  also  introduced  significant  clutter,  especially  when  employing  Fighter  Data  Link 
(FDL)  symbology  (Figure  1).  Second,  distributed  information  required  more  time  for  the  pilot  to  retrieve 
the  desired  information  (Reference  4).  Information  cannot  be  moved  within  the  current/existing  display 
FOV  without  creating  consistency  and  interpretation  problems.  AFRL/HECV  designed  a  symbology  to 
address  these  problems,  the  Non-Distributed  Flight  Reference  (NDFR). 

The  NDFR  allowed  direction  and  situation  awareness  to  be  maintained  while  the  pilot  was 
looking  off-boresight  during  tactical  maneuvering.  This  symbology  formed  a  composite  “information 
stamp”. 
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Figure  Is  Distributed  HMD  flight  reference  symbology  with  FDL 


The  USAF  TPS  conducted  testing  of  the  original  NDFR  symbology  during  the  Have  ATTITUDE 
project.  That  project  used  the  HMD  on  VISTA  to  gather  data  to  evaluate  and  verify  the  NDFR  in  the 
flight  environment  and  operational  context.  The  NDFR  was  compared  to  the  MIL-STD-1787C  HUD 
symbology,  which  was  used  as  a  baseline,  and  to  the  Visually  Coupled  Acquisition  Targeting  System 
(VCATS)  on  the  HMD.  Each  of  the  three  configurations  contained  own-ship  airspeed,  heading,  altitude, 
and  attitude  information.  The  three  test  symbology  configurations  were  evaluated  during  ground  and 
flight  tests.  Test  results  showed  that  the  NDFR  was  better  than  both  the  HUD  and  VCATS  symbology 
for  recall  of  own-ship  status,  unusual  attitude  recognition  time,  and  as  aids  during  operational  air-to-air 
(A/ A)  and  air-to-ground  (A/G)  tasks.  However,  Have  ATTITUDE  made  recommendations  for 
improvement  of  the  NDFR  symbology.  Revisions  were  made  by  AFRL/HECV  and  the  new  symbology 
was  designated  the  Advanced  NDFR  (ANDFR). 

This  project  used  the  HMD  on  VISTA  to  examine  two  different  configurations  of  the  ANDFR  in 
the  flight  environment  and  operational  context.  The  Baseline  (BL)  symbology  set  served  as  the  baseline 
for  comparison  and  includes  the  MIL-STD-1787C  HUD  distributed  airspeed  and  altitude  in  the  HMD. 
The  Baseline  Plus  (BL+)  configuration  added  the  Arc  Segmented  Attitude  Reference  (ASAR)  to  the  BL 
set  in  the  HMD.  The  other  configuration  contained  the  complete  ANDFR  symbology  set  without  any 
distributed  own-ship  status  information.  All  three  symbology  sets  were  generated  by  the  VISTA 
Simulation  System  (VSS)  and  projected  onto  the  HMD.  The  on-boresight  symbology  set  for  all  three 
configurations  was  the  MIL-STD-1787C  HUD. 

The  test  team  used  a  build-up  approach  to  evaluate  each  configuration  during  this  test.  First, 
own-ship  parameter  recall  was  accomplished,  followed  by  unusual  attitude  recoveries,  and  finally 
evaluation  in  an  operational  context.  Pilot  workload  and  situational  awareness  ratings  were  used  to 
assess  the  symbology  as  an  aid  in  the  piloting  tasks.  Aircrew  evaluated  potential  operational  utility  as 
well.  For  the  purpose  of  comparison  to  the  Have  ATTITUDE  test  results,  many  of  the  test  procedures 
and  evaluation  criteria  were  taken  from  the  Have  ATTITUDE  test  plan  (Reference  5). 


TEST  ITEM  DESCRIPTION 


Advanced  Non-Distributed  Flight  Reference  (ANDFR)  Symbology 

The  test  item  for  the  Have  SYCLOPS  test  program  was  the  ANDFR  symbology  set,  which  was  to 
be  compared  to  the  BL  and  BL+.  The  ANDFR  symbology  was  a  modification  of  the  original  NDFR 
symbology  and  was  developed  by  AFRL  to  further  optimize  own-ship  status  information,  thereby 
enhancing  the  pilot’s  ability  to  quickly  collect  and  process  information  from  the  HMD.  The  symbology 
combined  digital  flight  information  with  an  analog  attitude  reference  called  ASAR.  The  design 
attempted  to  de-clutter  and  localize  information  to  afford  space  in  the  HMD  FOV  for  targeting  or  other 
critical  cueing  symbology.  The  ANDFR  was  presented  high  in  the  pilot’s  FOV  for  A/G  operational  tasks 
and  low  in  the  pilot’s  FOV  for  A/A  operational  tasks. 

The  ANDFR  included  airspeed,  altitude,  flight  path  angle,  and  heading  information  presented 
digitally  around  a  line  drawing  in  the  shape  of  an  aircraft’s  wings  and  tail,  similar  in  design  to  a  HUD 
flight  path  marker  (Figure  2).  Airspeed  (480  KIAS)  was  presented  below  this  marker  on  the  left,  altitude 
(22600  ft)  was  presented  on  the  right,  flight  path  angle  (-12  degrees)  was  presented  in  the  center  of  the 
marker,  and  heading  (090  degrees)  was  presented  at  the  bottom  of  the  symbology,  below  the  ASAR. 

This  configuration  was  consistent  with  the  “Basic  T”  primary  flight  reference  layout,  with  attitude 
indicator  in  the  center,  airspeed  on  the  left,  altimeter  on  the  right,  and  heading  indicator  or  HSI  below 
(Reference  6).  The  aircraft  symbol  was  fixed  relative  to  the  HMD  FOV  and  the  ASAR  moved  around  it. 
Attitude  was  interpreted  by  comparing  the  aircraft  reference  symbol  to  the  attitude  symbology.  ANDFR 
used  the  ASAR  display  (Reference  7  &  8)  for  the  attitude  component  of  the  symbol  set  (Figure  3). 

Both  the  altitude  and  airspeed  displays  had  rolling  number  displays  for  all  digits,  similar  to  a  car 
odometer,  providing  trend  information  to  the  pilot.  The  airspeed  was  displayed  in  1-knot  increments,  and 
altitude  in  100  ft  increments  (note:  illustrations  and  figures  in  this  document  may  show  different  altitude 
increments;  however,  the  ANDFR  symbology  used  100  ft  increments). 
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The  ASAR  format  used  in  the  original  version  of  the  NDFR  symbology  was  modified  by  the 
USAF  as  an  HMD  attitude  reference  and  has  flown  on  both  the  X-31  (Reference  9)  and  on  an  AV-8B 
(Reference  10)  as  part  of  two  separate  flight  evaluations.  Pilots  reported  that  the  symbology  was  easy  to 
interpret  and  usable  for  global  attitude  maintenance.  Further  modifications,  including  the  non-distributed 
digital  information,  were  added  to  the  ASAR  during  the  Have  ATTITUDE  project. 

The  ANDFR  provided  own-ship  status  (aircraft  state)  information  intended  for  HMD  use  during 
off-boresight  viewing.  The  symbology  was  comprised  of  airspeed  and  altitude  indicators  in  a  digital 
odometer  format  (for  trend),  a  digital  heading  indicator,  digital  climb-dive  readout  (precision  attitude), 
and  an  ASAR.  The  ANDFR  was  designed  to  meet  the  deficiencies  noted  during  the  flight  test  evaluation 
of  the  NDFR  conducted  during  the  Have  ATTITUDE  evaluation.  The  flight  test  evaluation  of  the  NDFR 
concluded  that  off-axis  use  of  own-ship  status  symbology  such  as  the  NDFR  enhances  pilot  performance 
during  air-to-air  (A/ A)  and  air-to-ground  (A/G)  operational  tasks  over  that  of  HUD  use  alone  (Reference 
1 1).  However,  it  was  noted  that  the  NDFR  HMD  symbology  lacked  the  ability  to  sufficiently  convey 
rate-of-change  information  to  the  pilot  using  a  pure  digital  readout  for  the  airspeed  and  altitude 
indicators.  It  was  also  noted  that  the  prototype  format  of  the  ASAR  did  not  provide  enough  information 
to  determine  attitude  precision  concerning  climb-dive  or  roll  angles,  especially  near  straight  and  level 
flight. 

The  ANDFR  symbology  was  based  on  the  attitude  format  of  the  final  ASAR  design  developed  by 
Daimler-Benz  Aerospace  (Reference  7  &  8).  The  ASAR  attitude  symbology  conveyed  climb-dive  and 
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roll  using  a  180°  arc  segment  referenced  against  a  fixed  aircraft  symbol.  At  level  flight,  a  half  circle  was 
displayed  representing  a  180°  attitude  arc.  A  more  complete  circle  was  formed  when  diving  close  to  90° 
and  almost  no  arc  was  displayed  during  90°  climb.  Dots  and  gaps  along  the  attitude  arc  were  provided  to 
convey  ±  30°  and  ±  60°  climb-dive,  and  act  as  a  “fly  to”  attitude  reference.  Roll  angles  were  given  by  the 
angular  relationship  between  the  fixed  aircraft  symbol  and  the  center  of  the  attitude  arc  rotating  about  the 
aircraft  symbol  (Figure  3). 

In  an  effort  to  increase  the  precision  of  the  attitude  information  conveyed,  a  digital  flight  path 
angle  indicator  was  added  to  the  center  of  the  aircraft  symbol.  The  remaining  own-ship  status 
information  was  laid  out  according  to  the  “Basic  T”  instrument  configuration  with  the  digital  odometer 
airspeed  indicator  under  the  left  wing  of  the  aircraft  symbol,  the  digital  odometer  altitude  indicator  under 
the  right  wing,  and  a  truncated  (first  two  digits)  digital  heading  located  at  the  bottom  of  the  arc 
underneath  the  ground  pointer. 

Baseline  and  Baseline  Plus  HMD  Off-boresight  Symbol  Sets 

The  BL  symbol  set  contained  an  airspeed  and  altitude  indicator  identical  in  format  to  the  MTT 
STD-1787C  HUD  symbology.  The  BL+  symbol  set  contained  the  same  airspeed  and  altitude  indicators 
found  in  the  BL  format,  but  added  the  MIL-STD-1787C  HUD  aircraft  heading  tape  and  the  ASAR. 

The  three  symbology  sets  for  the  test  (Figure  4)  were  adapted  for  VISTA  by  Veridian 
Engineering  with  consultation  from  AFRL/HECV.  The  software  was  loaded  into  the  VSS  onboard 
computers,  which  displayed  it  into  the  HMD.  The  evaluation  pilot  was  in  the  front  seat  of  the  VISTA 
with  the  HMD. 
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Figure  4:  BL,  BL+,  and  ANDFR  HMD  Symbologies 


HUD  Symbology 

The  M1L-STD-1787C  HUD  symbology  was  included  to  serve  as  an  on-boresight  baseline  to 
compare  the  relative  benefit  of  the  HMD  symbology  sets.  This  symbology  included  a  pitch  ladder,  flight 
path  marker,  and  bank  scale  for  own-ship  attitude  reference,  airspeed  and  altitude  indicators,  and  a 
heading  tape.  Both  the  airspeed  indicator  and  altimeter  contained  a  digital  readout  with  a  dial  and 
counter  pointer  (Figure  5). 
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Figure  5:  MIL-STD-1787C  HUD 


TEST  OBJECTIVES 

The  overall  test  objective  was  to  determine  if  the  addition  of  the  off-boresight  ANDFR 
symbology,  containing  own-ship  status  information,  to  an  HMD  aids  in  operationally  representative,  off- 
boresight,  air-to-air  and  air-to-ground  piloting  tasks. 

Both  ground  and  flight  test  were  accomplished  to  reach  this  objective.  The  VISTA  ground 
simulation  mode  was  used  for  pilot  symbology  and  HMD  system  familiarization.  This  ground  simulation 
mode  was  also  used  to  perform  information  recall  ground  tests,  unusual  attitude  recoveries,  A/A  all 
aspect  missile  defense  tests,  and  A/G  pop  attacks  using  the  BL,  BL+,  and  ANDFR  symbologies.  Flight 
tests  were  accomplished  to  compare  the  symbologies  during  actual  unusual  attitude  recoveries  and  during 
A/A  and  A/G  operationally  representative  tasks. 
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TEST  AND  EVALUATION 


GENERAL 

The  overall  test  objective  was  met  through  the  successful  accomplishment  of  all  the  supporting 
tests.  Three  off-boresight  helmet-mounted  display  (HMD)  symbology  sets  intended  for  use  in  fixed-wing 
aircraft  were  evaluated  using  the  NF-16D  Variable-stability  In-flight  Simulator  Test  Aircraft  (VISTA). 
Two  of  the  symbol  sets  were  variations  of  a  newly  modified  off-boresight  flight  reference  symbology, 
called  the  Advanced  Non-Distributed  Flight  Reference  (ANDFR).  Testing  was  performed  at  the  USAF 
Flight  Test  Center  at  Edwards  AFB,  California,  by  the  Have  SYCLOPS  test  team  from  the  USAF  Test 
Pilot  School  (USAF  TPS).  Ground  tests  were  conducted  from  9  to  14  April  2003  and  flight  tests  from  10 
to  23  April  2003. 

Six  VISTA  ground  simulation  tests  were  accomplished  for  data  collection.  Two  calibration  and 
twelve  VISTA  flight  test  sorties  totaling  19.3  hours  were  flown  in  support  of  this  project.  Three  T-38 
target  sorties  were  required,  totaling  3.5  flight  hours. 

The  VISTA’s  data  acquisition  system  (DAS)  was  used  to  record  data.  Testing  was  performed  by 
three  evaluation  pilots  (EPs),  two  evaluation  flight  test  engineers  (EFs),  and  one  evaluation  flight  test 
navigator  or  weapon  systems  officer  (EN).  All  evaluators  provided  comments  and  ratings.  Veridian 
Flight  Research  performed  specialty  engineering. 

Ground  testing  and  calibration  flights  were  accomplished  prior  to  flight  test  to  ensure  that  VISTA 
Simulation  System  (VSS)  was  correctly  programmed  and  functioning  properly  for  the  ground/flight  test 
configurations.  The  three  off-boresight  HMD  symbology  configurations  were  Baseline  (BL),  utilizing 
the  MIL-STD-1787C  HUD  airspeed  and  altitude  indicators;  Baseline  Plus  (BL+),  which  added  the  Arc 
Segmented  Attitude  Reference  (ASAR)  analog  display  and  MIL-STD-1787C  HUD  heading  tape  to  the 
BL  set;  and  the  complete  ANDFR. 

Helmet  Tracker 

The  helmet  tracker  used  for  determining  helmet  pointing  angles  malfunctioned  during  flight 
testing  and  failed  to  provide  reliable  and  precise  information.  As  a  result,  the  designed  occlusion  or 
“blanking”  of  HMD  symbology  when  looking  on-boresight  through  the  HUD  did  not  occur  during  any 
flight  test  tasks.  The  impact  was  found  to  be  minimal  by  the  test  team.  Off-boresight  times  were 
consistently  large  enough  that  the  few  times  the  evaluators  did  look  at  the  HUD,  they  remarked  that  the 
distraction  was  minimal.  In  addition,  most  evaluators  mentioned  that  occlusion  would  probably  not  have 
occurred  in  any  case  because  their  head  did  not  come  all  the  way  back  to  within  the  HUD  field  of  view 
(FOV).  They  found  themselves  glancing  back  into  the  HUD  while  their  head  was  still  pointed  about  15° 
to  20°  off-boresight.  Therefore,  qualitative  information  from  transition  to  and  from  on-boresight  was 
considered  valid.  To  compensate  for  the  unreliable  and  imprecise  helmet  tracking  data,  evaluators 
estimated  their  percent  time  off-boresight,  considering  glances  at  the  HUD  to  be  “on-boresight”.  When 
necessary,  these  estimates  were  confirmed  using  the  video  tapes  from  the  HMD  and  HUD  displays. 


INFORMATION  RECALL  TEST 


Compare  the  aircrew’s  ability  to  quickly  interpret  the  BL,  BL+,  and  ANDFR  symbologies. 
Test  Procedures; 


This  test  was  intended  to  replicate  a  test  conducted  by  the  Have  ATTITUDE  test  team  for  the 
different  symbology  configurations  (Reference  1)  and  was  accomplished  in  VISTA  in  the  ground 
simulation  mode.  Prior  to  data  collection,  the  evaluators  performed  a  series  of  simulated  roll,  loop,  and 
barrel  roll  maneuvers  which  allowed  evaluators  to  familiarize  themselves  with  the  symbology  sets  before 
using  them  for  test  points. 

All  evaluators  began  the  ground  test  with  the  MIL-STD-1787C  HUD  symbology  which  was  used 
as  the  baseline  for  performance  comparison.  Due  to  the  previously  noted  problems  with  the  helmet 
tracker,  the  virtual  HUD  symbology  could  not  be  properly  displayed  in  the  HMD.  The  visor  cover  was 
not  worn  (deviation  from  the  test  plan)  and  the  evaluators  used  the  actual  HUD  as  the  on-boresight 
reference.  The  evaluators  assessed  that  the  external  visual  clutter  (hanger  doors,  etc)  did  not  affect  their 
performance  during  this  task.  For  the  HUD  symbology,  all  evaluators  began  by  looking  straight  ahead. 
For  the  BL+  and  ANDFR  symbologies,  all  evaluators  accomplished  the  tests  with  their  heads  pointed 
approximately  45°  off-boresight.  When  the  evaluator  called  “ready,”  the  VSS  operator  displayed  a 
predetermined  aircraft  attitude  in  the  HMD  or  HUD.  The  symbology  was  displayed  for  500  milliseconds. 
The  evaluator  verbally  reported  heading,  airspeed,  altitude,  bank,  and  flight  path  angle.  These  parameters 
were  recorded  by  the  VSS  operator  and  on  the  VISTA  data  tape.  Six  aircraft  attitudes  were  presented  in 
the  same  sequence  for  each  evaluator.  The  same  procedure  was  repeated,  with  different  aircraft  attitudes, 
using  the  BL+  and  the  ANDFR  symbologies. 

The  evaluator’s  verbal  comments  were  recorded  during  and  after  each  test  point.  The  evaluator 
completed  a  questionnaire  sheet  immediately  following  the  ground  test  session.  The  verbal  comments 
and  the  post-test  questionnaire  comments  were  reviewed  and  summarized  in  a  daily  ground  test  report 
(Reference  12). 

Test  Results: 


This  task  was  successful  in  supporting  the  overall  test  objective.  The  evaluator  responses  for  the 
five  information  categories:  heading,  airspeed,  altitude,  bank,  and  flight  path  angle  (FPA)  were  scored 
based  on  Table  1.  The  collected  data  and  response  scores  are  tabulated  for  each  evaluator  in  Appendix  C. 


Table  1:  Evaluator  Recall  Scoring  for  Each  Information  Category 


Information  Categories 

Recall  Score 

Heading 

Airspeed 

Altitude 

Bank 

FPA 

Satisfactory-  2 

+/-  10  deg 

Correct  hundreds 
and  tens  digit 

Correct  thousands 
and  hundreds  digit 

+/-  15  deg 

+/-  15  deg 

Marginal-  1 

+/-  30  deg 

Correct  hundreds 
digit 

Correct  thousands 
digit 

+/-  30  deg 

+/-  30  deg 

Unsatisfactory-  0 

>  30  deg 

Incorrect  hundreds 
digit 

Incorrect 
thousands  digit 

>  30  deg 

>  30  deg 
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The  average  pilot  score  for  the  analog  information  categories  (bank  and  FPA)  are  shown  in  Figure 
6.  The  average  scores  using  BL+  or  ANDFR  for  bank  and  FPA  responses  were  higher  than  the  average 
scores  for  the  HUD.  However,  the  average  score  for  FPA  using  BL+  was  slightly  higher  than  that  for 
ANDFR.  Although  FPA  is  depicted  as  an  analog  parameter,  it  was  also  available  digitally  in  BL+  and 
ANDFR.  The  digital  precision  for  FPA  located  near  the  center  of  the  display  could  account  for  the 
higher  average  scores  using  BL+  and  ANDFR.  The  average  pilot  score  for  digital  information  categories 
(heading,  airspeed,  and  altitude)  are  shown  in  Figure  7.  The  average  scores  for  heading,  airspeed,  and 
altitude  responses  were  low  and  did  not  vary  significantly  between  the  three  symbologies.  Heading  was 
rarely  recalled  correctly  if  noticed  at  all.  The  average  scores  for  altitude  using  BL  or  BL+  were  slightly 
higher  than  that  of  ANDFR,  but  the  average  score  for  airspeed  using  ANDFR  was  higher  than  that  of  BL 
or  BL+.  Histograms  of  non-pilot  recall  data  are  located  in  a  separate  data  package  (Reference  12). 
Overall,  all  pilots  recalled  analog  information  more  precisely  and  more  often  than  they  did  digital 
information. 
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Pilot  Attitude  Recall  -  Analog  Recall  Scores  Averaged  Across  Aircrew 
Have  SYCLOPS  Aircraft:  NF-16B  VISTA,  86-0048 
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Figure  6:  Analog  Information  Category  Recall  Average  Scores 
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Figure  7:  Digital  Information  Category  Recall  Average  Scores 


All  possible  combinations  of  information  categories  that  received  the  highest  score  (2)  for  all 
categories  are  also  shown  in  Appendix  C.  A  “Mean  Correct  Response  Frequency”  (MCRF)  for  each 
symbology  was  computed  by  dividing  the  number  of  correct  information  category  combinations  by  the 
total  possible  information  category  combinations.  This  parameter  was  developed  and  used  by  AFRL  for 
analysis  of  parameter  recall  data  collected  from  laboratory  tests  (Reference  4).  The  MCRF  for  each  of 
the  symbologies  is  plotted  in  Figure  8.  When  recalling  static  aircraft  parameters,  pilots  were  able  to 
correctly  recall  precise  data  more  often  when  using  ANDFR  than  with  the  HUD  or  BL+.  Likewise,  pilots 
were  able  to  correctly  recall  precise  data  more  often  using  BL+  than  with  the  HUD. 

ANDFR  was  the  preferred  symbology  for  parameter  recall.  A  typical  pilot  comment  was, 
“ANDFR  was  the  easiet  for  seeing  multiple  pieces  of  information  due  to  their  close  proximity.” 

However,  pilots  were  still  only  able  to  focus  on  one  to  two  parameters  in  the  short  amount  of  time.  The 
analog  pitch  and  bank  information  was  easier  to  perceive  using  the  Arc  Segmented  Attitude  Reference 
(ASAR)  without  focusing  specifically  on  those  parameters.  A  pilot  could  focus  on  a  digital  parameter 
and  sense  an  analog  parameter  with  his  peripheral  vision.  Pilots  concluded  that  interpreting  bank  and 
pitch  from  the  ASAR  was  easier  when  the  aircraft  was  in  a  nose  low  attitude.  However,  the  bank 
information  was  identified  as  being  somewhat  difficult  to  interpret  in  nose  high  attitudes.  A  better 
horizon  reference  would  improve  the  quick  recognition  of  bank  angle  at  any  flight  path  angle.  The  off- 
boresight  horizon  reference  should  be  improved  (Rl)\  Adding  a  horizon  bar  (similar  to  the  HUD) 
that  moves  up  and  down  and  rotates  relative  to  the  digital  FPA  circle  is  one  possible  solution  (Appendix 


Numerals  preceded  by  an  R  within  parentheses  at  the  end  of  a  sentence  correspond  to  the 
recommendation  numbers  tabulated  in  the  Conclusions  and  Recommendations  section  of  this  report. 
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Number  of  Information  Categories  Recalled 

Figure  8:  Parameter  Recall  Mean  Correct  Response  Frequency  (MCRF) 


The  BL+  performance  was  very  similar  to  ANDFR,  but  the  distributed  nature  of  the  altitude  and 
airspeed  information  made  quickly  recalling  multiple  parameters  more  difficult.  The  existing  distribution 
of  the  BL+  symbology  was  not  objectionable,  since  quick  parameter  recognition  during  operational  tasks 
usually  involves  recognizing  parameters  one  at  a  time  in  a  certain  order  of  importance  (i.e.  bank,  pitch, 
then  altitude).  Recalling  more  than  two  information  categories  (one  precise  and  one  approximate)  with  a 
0.5  second  flash  of  HUD  information  was  extremely  difficult  due  to  the  widely  distributed  amount  of 
information.  Using  BL+  or  the  HUD  would  require  more  scan  time  to  collect  the  same  amount  of 
information  gathered  using  ANDFR. 


UNUSUAL  ATTITUDE  RECOVERIES 

Compare  the  BL,  BL+,  and  ANDFR  symbology  sets  as  aids  in  recovering  from  unusual 
attitudes  while  initially  looking  off-boresight. 

Test  Procedures: 


The  test  team  performed  unusual  attitude  (UA)  recoveries  in  ground  simulation  and  in  flight  using 
BL,  BL+  and  ANDFR.  The  visor  cover  was  not  worn  (deviation  from  test  plan)  over  the  HMD  visor. 
This  was  done  so  the  actual  HUD  symbology  was  available  during  the  tests  for  attitude  information  if  the 
evaluator  needed  to  transition  to  the  HUD  for  unusual  attitude  recognition  and/or  recovery. 

Consequently,  the  outside  horizon  was  available  during  flight  tests  as  an  additional  aid  in  determining 
attitude  information. 

All  evaluators  began  the  ground  and  flight  test  points  with  an  approximate  90°  off-boresight  head 
position.  When  the  evaluator  called  “ready”,  the  VSS  operator  then  started  the  VSS  masking  maneuver 
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sequence.  After  the  masking  maneuver  was  complete,  the  VSS  sounded  a  tone,  simultaneously  displayed 
the  appropriate  symbology  depicting  the  unusual  attitude  initial  conditions,  and  started  recording  time 
and  control  inputs.  At  the  tone,  the  evaluator  was  able  to  view  the  symbology  presented  and  use  it  to 
recover  from  the  displayed  unusual  attitude.  Recovery  was  initiated  off-boresight  and  completed  looking 
on-boresight.  The  VSS  control  laws  were  used  in  VISTA  for  unusual  attitude  recovery  flight  tests 
(Appendix  B). 

Speed  and  correctness  of  the  evaluator’s  first  significant  control  inputs  were  recorded  and 
compared.  The  evaluator’s  verbal  comments  were  recorded  during  and  after  each  test  point.  The 
evaluator  completed  a  questionnaire  sheet  (Appendix  G)  for  each  symbology  and  a  daily  test  report 
following  the  ground  test  session  and  flight  test  sortie.  The  verbal  comment  tapes  and  questionnaire 
comments  were  summarized  in  the  daily  ground/flight  test  reports  (Reference  12). 

Test  Results: 


The  test  objective  was  met.  Quantitative  ground  simulation  data  for  time  to  first  significant  input 
was  not  available  due  to  data  dropouts.  However,  ground  simulation  qualitative  comments  and  flight  test 
data  were  collected  from  all  three  EPs.  Non-pilot  unusual  recovery  data  are  located  in  Reference  12. 

The  team  tabulated  the  percentage  of  correct  initial  recovery  inputs  and  the  mean  time  to  correct  initial 
recovery  inputs  for  both  ground  simulation  and  flight  tests  (Figures  9).  Figure  10  displays  the  mean  time 
to  first  significant  input  for  incorrect  inputs. 

Figure  9  shows  the  mean  times  to  first  significant  pilot  input.  The  mean  time  to  first  correct 
significant  input  was  1.42  seconds  with  ANDFR,  1.44  seconds  with  BL+,  and  1.61  seconds  with  BL. 
However,  the  standard  deviations  for  each  symbology  varied  a  lot.  A  statistical  95%  confidence  level 
student’s  T-test  was  conducted  on  the  times  and  correctness  of  recoveries.  The  test  team  could  not 
conclude  with  95%  confidence  that  the  mean  time  to  first  significant  input  using  ANDFR  was  less  than 
those  with  BL  and  BL+.  Likewise,  the  test  team  could  not  conclude  with  95%  confidence  that  the 
correctness  of  the  inputs  using  ANDFR  was  better  than  BL  and  BL+.  See  Appendix  D  for  statistical 
computations.  Overall,  ANDFR  symbology  was  not  better  than  BL  or  BL+  with  respect  to  mean  time  to 
first  significant  control  input  or  correctness  of  the  input. 


Pilot  Unusual  Attitudes  -  Mean  Time  to  1st  Significant  CORRECT  Input 
Have  SYCLOPS  Aircraft:  NF-16D  VISTA,  86-0048 

Flight  Test:  1 0-23  April  2003  Edwards  AFB,  CA 
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Figure  9.  Mean  Time  to  First  Significant  Input  for  Correct  UAR  in  Flight 


Pilot  Unusual  Attitudes  -  Mean  Time  to  1st  Significant  INCORRECT  Input 


Have  SYCLOPS  Aircraft:  NF-16D  VISTA,  86-0048 

Flight  Test:  10-23  April  2003  Edwards  AFB,  CA 
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Figure  10.  Mean  Time  to  First  Significant  Input  for  Incorrect  UAR  in  Flight 


Based  on  pilot  questionnaire  results,  unusual  attitude  parameters  could  be  rapidly  recognized 
using  either  the  BL+  or  ANDFR  symbologies.  The  Arc  Segmented  Attitude  Reference  (ASAR) 
contained  in  both  BL+  and  ANDFR  was  the  primary  reason  pilots  obtained  similar  recognition  and 
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recovery  performance  with  the  two  symbologies.  Required  control  inputs  for  UA  recovery  were  easily 
determined  once  the  ASAR  symbology  was  viewed,  and  confidence  in  spatial  orientation  was  high 
before,  during,  and  after  recovery  using  either  BL+  or  ANDFR.  The  pilots  were  generally  more 
confident  of  their  spatial  orientation  throughout  the  entire  UA  recognition  and  recovery  when  the  ASAR 
symbology  was  available. 

The  BL  and  BL+  digital/analog  dials  were  intuitive  and  required  less  adjustment  for  all  pilots  due 
to  HUD  training  paradigms.  The  heading  “bar”  was  unanimously  deemed  unnecessary  for  a  UAR.  The 
BL/BL+  digital/analog  airspeed/altitude  dials  provided  better  information  to  the  pilots  than  the  ANDFR 
digital  odometer  and  were  especially  helpful  in  identifying  extreme  values  and  trends  (rates)  due  to  the 
“spinning”  or  movement  of  the  dials.  These  spinning  dials  were  the  only  way  the  BL  symbology  would 
cue  a  pilot  that  he  was  in  an  unusual  attitude  while  off-boresight. 

The  BL  required  transitioning  to  the  HUD  for  UA  recognition/confirmation  and  recovery.  Even 
though  the  outside  horizon  was  available,  determining  aircraft  flight  path  angle  was  difficult  due  to  the 
disorienting  masking  maneuvers  and  lack  of  an  off-boresight  attitude  reference  symbology.  If  BL  is  used 
as  the  default  symbology,  recognition  of  unusual  attitudes  while  looking  off-boresight  may  be  delayed. 
Addition  of  the  ASAR  format  to  the  BL  symbology  set  (as  in  BL+)  would  help  prevent  unrecognized 
unusual  attitudes. 

The  BL+  was  preferred  by  all  pilots.  The  ASAR  feedback  during  recovery  contributed  to  the 
evaluator’s  quick  recognition  and  correction  of  initial  incorrect  inputs.  The  ASAR  was  especially  strong 
in  graphically  communicating  the  unusual  attitude  during  nose-low  conditions.  The  evaluator  simply 
needed  to  roll  and  pull  towards  the  opening  of  the  ASAR  which  enabled  the  evaluator’s  ability  to  recover 
from  a  nose  low  unusual  attitude  completely  off-boresight  while  maintaining  situation  awareness  during 
mission  tasks/target  search.  The  only  consistent  deficiency  of  ASAR  is  the  non-intuitive  presentation  of 
nose-high  unusual  attiudes.  The  evaluator  could  not  simply  roll  and  pull  to  the  opening  of  the  ASAR  and 
at  times  would  roll  in  the  wrong  direction  initially. 


AIR-TO-AIR  TASK 

Compare  the  BL,  BL+,  and  ANDFR  symbology  sets  as  aids  for  off-boresight  air-to-air 
operational  piloting  tasks. 

Medium  Altitude  All-Aspect  Missile  Defense  Test  Procedures: 

Pilot  workload  and  situation  awareness  ratings  were  measured  while  maintaining  A/C  parameters 
within  bounds  during  an  all-aspect  missile  defense  maneuver  or  “notch”.  The  task  was  performed  using 
BL,  BL+,  and  ANDFR  symbologies. 

Ground  and  flight  tests  were  conducted  by  three  different  EPs.  Ground  tests  were  accomplished 
using  the  VISTA  VSS  system  to  display  a  simulated  target  (target  designator  box  1 1  nm  in  front  of 
VISTA)  with  a  representative  relative  movement  of  an  airborne  target  during  the  maneuver.  Flight  tests 
were  conducted  using  a  T-38  as  the  target  aircraft.  Evaluators  attempted  to  maximize  off-boresight 
visual  search  and  maintenance  time. 

For  flight  testing,  two  predetermined  waypoints,  20  nm  apart,  served  as  anchor  points  for  the  test 
and  target  aircraft.  After  both  aircraft  called  “ready”  the  test  aircraft  called  “turn  in.”  The  target 
aircraft  s  initial  airspeed  and  altitude  was  300  KIAS  at  18000  ft  MSL.  The  test  aircraft’s  initial  airspeed 
and  altitude  was  300  KIAS  at  16000  ft  MSL.  At  “turn  in”,  the  two  aircraft  turned  toward  each  other  and 
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the  test  aircraft  began  a  radar  search  to  acquire  a  lock.  This  was  seldom  successful  because  the  fire 
control  radar  in  the  VISTA  wasn’t  working  correctly  and  would  shut  itself  down  after  approximately  5 
minutes  of  use.  If  available,  both  aircraft  used  air-to-air  TACAN  for  range  information. 

At  10  nm  separation,  the  evaluator  called  “action”  over  the  radio  and  began  a  defensive  “notch” 
maneuver  (putting  the  adversary  aircraft  at  90°  off  heading)  based  primarily  on  VISTA’s  radar  range  and 
secondarily  on  air-to-air  TACAN.  See  Figure  1 1  for  an  illustration  of  the  “notch”  maneuver.  If  both 
radar  range  and  air-to-air  TACAN  were  not  available,  a  10  nm  call  from  SPORT  (AFFTC  Air  Traffic 
Control)  was  used  to  initiate  the  “notch.”  At  “action,”  the  evaluator  executed  an  approximately  4  to  6  g, 
oblique  turn  to  approximately  30°  to  60°  nose  low,  putting  the  target  aircraft  approximately  90°  off 
heading.  The  test  aircraft  descended  to  the  block  altitude  of  10000  to  14000  ft  MSL  and  accelerated  to 
400  KIAS.  The  evaluator's  goal  was  to  descend  to  the  simulated  “floor”  while  achieving  a  simulated 
tactical  airspeed,  maintaining  a  90°  off-boresight  angle  to  the  target  aircraft,  and  visually  searching  for 
the  target  aircraft.  The  simulated  ground  level  was  10000  ft  MSL  and  the  maximum  allowable  airspeed 
was  450  KIAS  due  to  the  Viper  IV  helmet's  ejection  limit.  The  evaluation  continued  if  the  test  aircraft 
descended  below  10000  ft  MSL,  but  the  overall  performance  rating  was  graded  as  “unsatisfactory”.  The 
maneuver  was  terminated  if  the  aircraft  descended  below  9000  ft  MSL. 

The  safety  pilot  (SP)  armed  the  VISTA’s  instrumentation  system  timing  prior  to  the  action  point. 
VISTA  recorded  total  off-boresight  helmet  time  during  the  engagement.  Timing  began  once  the  bank 
angle  was  reduced  to  less  than  20°  during  the  roll  out  from  the  initial  “action”  turn.  The  safety  pilot 
called  “terminate”  and  stopped  the  VISTA’s  instrumentation  system  recording  60  seconds  after  “action” 
or  at  intercept,  whichever  occurred  first. 

At  the  "action"  call,  the  target  aircraft  descended  to  15000  ft  MSL,  maintained  300  KIAS  and 
turned  20°  into  the  test  aircraft  every  10  seconds  until  visually  acquiring  the  target.  Once  the  target  was 
in  sight  with  the  test  aircraft,  the  target  aircraft  remained  behind  the  VISTA  until  intercept  or  the 
“terminate”  call. 

After  the  task,  the  evaluator  assigned  a  MCH  rating  and  CLSA  rating  using  the  scales  in 
Appendix  A.  The  MCH  rating  was  based  on  the  following  criteria: 

Desired:  Altitude:  10000  to  11000  feet  for  at  least  15  seconds 

Airspeed:  400  KIAS  ±20  KIAS  by  “terminate” 

Adequate:  Altitude:  10000  to  12000  feet  for  at  least  15  seconds 

Airspeed:  400  KIAS  ±40  KIAS  by  “terminate” 

Unsatisfactory:  Failure  to  meet  adequate  criteria 


Percent  time  off-boresight  was  also  recorded  for  comparison  and  could  be  measured  as  desired, 
adequate  or  unsatisfactory.  However,  it  was  not  used  to  determine  the  task  performance  rating.  Off- 
boresight  visual  time: 


Desired: 

Adequate: 


85%  off-boresight  visual  time 
65%  off-boresight  visual  time 


Figure  11:  All- Aspect  Missile  Defense  Maneuver 


The  evaluator  verbal  comments  and  ratings  were  recorded  during  and  after  each  test  point.  Each 
evaluator  completed  a  questionnaire  sheet  immediately  following  the  tasks  (ground  tests)  or  sortie  (flight 
tests),  and  completed  a  daily  flight  report  following  the  flight  test  sortie.  The  verbal  comment  tapes  and 
the  post-test  point  questionnaire  comments  were  included  in  the  daily  test  reports. 

Medium  Altitude  All-Aspect  Missile  Defense  Test  Results: 

Each  pilot  completed  three  tasks  per  symbology  set  for  a  total  of  nine  (the  EF  completed  eight 
total  tasks).  However,  due  to  the  lack  of  proficiency  in  flying  the  maneuver,  the  evaluators  felt  the  data 
from  the  first  three  tasks  of  each  test  sortie  were  invalid.  Therefore,  only  data  from  the  last  six  tasks  (two 
tasks  per  symbology  set  for  each  evaluator)  were  considered  valid  and  analyzed  for  results.  The  pilot  test 
results  are  shown  in  Figures  12  through  15. 

Figure  12  is  a  histogram  of  all  MCH  ratings  given  by  the  EPs  after  each  AAMD  task.  The  ratings 
are  divided  according  to  HMD  symbology  used,  and  each  histogram  bar  represents  the  total  number  of 
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tasks  receiving  that  rating.  MCH  ratings  were  assigned  based  on  the  mental  workload  and  overall 
performance  during  each  task.  Each  AAMD  task  was  assigned  an  overall  performance  rating  of  desired, 
adequate,  or  unsatisfactory,  based  on  airspeed  and  altitude  maintenance.  Lower  MCH  rating  numbers 
were  better.  Tasks  accomplished  successfully  with  acceptable  mental  workload  received  ratings  of  1 
through  3.  Tasks  accomplished,  but  only  with  an  unacceptably  high  workload  received  ratings  of  4 
through  6.  Ratings  of  7  through  9  described  a  task  that  was  not  accomplished  without  errors  and  without 
major  difficulty.  A  task  that  was  impossible  to  accomplish  reliably  received  a  rating  of  10.  Figure  13  is 
a  histogram  of  all  CLSA  ratings  given  by  the  EPs  after  each  AAMD  task.  Lower  CLSA  ratings  were 
better.  See  Appendix  A  and  Reference  13  for  more  information  on  the  MCH  and  CLSA  scales  used 
during  testing.  Percent  time  off-boresight  was  also  recorded  for  comparison  and  had  its  own  desired  and 
adequate  criteria.  However,  it  was  not  used  to  determine  the  task  performance  rating.  Figure  15  displays 
task  performance  ratings  versus  the  tasks  average  percent  time  off-boresight  for  each  symbology. 

MCH  and  CLSA  scales  are  subjective  evaluations  of  pilot  workload,  task  performance  and 
situation  awareness.  Therefore,  objective  statistical  analysis  was  not  used  to  determine  conclusions. 
However,  subjective  conclusions  were  drawn  from  the  data  and  pilot  comments. 
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Pilot  All-Aspect  Missile  Defense  -  Modified  Cooper  Harper  Ratings  Across  All  Pilots 
HaveSYCLOPS  Aircraft:  NF-16D  VISTA,  86-0048 

Flightiest:  10- 23  April  2003  Edwards  AFB,  CA 
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Figure  12:  Modified  Cooper-Harper  (MCH)  Ratings  for  AAMD  Tasks 
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Figure  13:  China  Lake  Situational  Awareness  (CLSA)  Ratings  for  AAMD  Tasks 


Figure  14:  Percentage  of  Time  Spent  Looking  Off-Boresight  During  AAMD  Tasks 
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Overall,  ANDFR  was  found  to  be  less  beneficial  than  BL  and  BL+  during  AAMD  tasks.  AAMD 
tasks  accomplished  using  the  ANDFR  symbology  resulted  in  similar  performance  and  situation 
awareness  as  those  tasks  accomplished  with  BL  or  BL+.  However,  the  mental  workload  required  to 
achieve  the  similar  performance  when  using  the  ANDFR  was  higher.  Pilots  had  to  force  themselves  to 
use  the  airspeed  and  altitude  odometers  and  it  took  more  time  to  interpret  the  displayed  information.  The 
pilots  spent  a  large  amount  of  time  off-boresight,  but  they  spent  more  time  focusing  on  the  ANDFR  and 
flying  the  maneuver  like  an  instrument  procedure  than  they  did  looking  for  the  actual  target.  Pilots 
would  rather  spend  most  of  the  time  searching  for  the  target  while  being  able  to  quickly  recognize  and 
interpret  the  dynamic  aircraft  parameter  information  being  displayed.  Although  the  ANDFR  performed 
well  for  quickly  and  accurately  recognizing  static  information  (see  Information  Recall  Test),  the  dynamic 
(trend)  data  displayed  with  the  airspeed  and  altitude  odometers  required  too  high  a  mental  workload  and 
too  much  time  for  interpretation. 

For  a  daytime  scenario  with  a  good  horizon,  pilots  preferred  the  BL  set  for  use  during  all-aspect 
missile  defense  maneuvers.  The  BL  symbology  was  the  easiest  for  transitioning  from  the  HUD  to  off- 
boresight.  Pilots  liked  the  similarity  between  the  HUD  and  BL  airspeed  and  altitude  displays.  The  off- 
boresight  analog  dials  gave  the  pilot  an  exceptional  sense  for  rate  of  change  in  altitude  and  airspeed.  In 
addition  to  the  analog  trend  data,  the  outside  horizon  was  useful  for  determining  relative  nose  low 
attitudes  and  how  much  to  pull  and  smooth  out  the  transition  to  level  flight  at  the  simulated  ground  level. 
Once  leveled  off  in  the  desired  altitude  block,  the  analog  dials  provided  sufficient  trend  data  to  allow 
pilots  to  spend  the  majority  of  off-boresight  time  looking  for  the  target.  This  required  minimal  workload 
from  the  pilot  and  allowed  for  maximum  situation  awareness  (SA). 

Pilots  concluded  that  the  BL  was  satisfactory  for  medium  altitude  AAMD  tasks  with  a  good 
horizon.  However,  AAMD  maneuvers  in  combat  will  most  often  be  performed  close  to  the  actual 
ground.  The  three  symbology  sets  evaluated  did  not  include  any  radar  altitude  displays.  The  pilots  still 
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need  to  look  out  front  in  order  to  clear  their  flight  path  of  any  terrain  or  obstacles.  However,  the  addition 
of  an  off-boresight  radar  altitude  display  would  aid  in  determining  proximity  to  terrain  and  allow  the  pilot 
to  spend  more  time  off-boresight  looking  for  the  threat  which  will  especially  aid  in  flight  over  level 
terrain.  A  radar  altitude  display  should  be  added  to  the  off-boresight  display  (R2). 

The  BL+  symbology  was  the  second  most  preferred  symbology.  The  similarity  to  the  HUD  in 
terms  of  the  altitude  and  airspeed  cues  was  preferred  as  with  the  BL  set.  However,  the  BL+  attitude 
display,  or  ASAR,  may  aid  pilots  during  A  AMD  maneuvers  performed  at  night  or  without  a  discemable 
horizon.  Attempts  to  remain  at  a  level  altitude  were  difficult  to  perform  because  quickly  interpreting 
flight  path  angles  within  approximately  20°  of  level  flight  was  more  difficult  when  looking  off-boresight. 
The  ASAR  flight  path  angle  (FPA)  resolution  in  shallow  climbs  or  dives  was  marginal  which  caused 
pilots  to  focus  on  the  digital  FPA  indication.  This  required  more  time  to  read  and  interpret  the  off- 
boresight  display  and  pilots  found  it  easier  to  quickly  glance  in  the  HUD  to  check  the  on-boresight 
display  of  the  flight  path  marker  and  horizon  line.  The  pilots  liked  the  off-boresight  digital  display  of 
FPA  for  precision.  However,  for  quick  cross-checks  to  ensure  the  aircraft  was  not  descending,  pilots 
preferred  an  analog  display  for  comparison  of  the  flight  path  marker  (circle  with  digital  number  inside)  to 
the  horizon  line  similar  to  the  HUD.  The  off-boresight  analog  flight  path  angle  reference  should  be 
improved  for  quicker  interpretation  of  flight  path  angles  less  than  20°  (R3).  A  suggested  solution 
for  improving  the  off-boresight  horizon  (discussed  previously)  and  improving  the  analog  FPA  reference 
is  located  in  Appendix  H. 


The  BL+  off-boresight  aircraft  heading  display  was  never  needed  and  was  so  large  that  the 
display  became  cluttered  and  caused  distraction.  The  pilots  noted  a  potential  benefit  from  having 
helmet  heading”  available  off-boresight  instead  of  aircraft  heading.  A  display  of  helmet  heading  is 
much  more  valuable  to  a  pilot  for  use  in  acquiring  targets  visually  from  either  communications  or  off- 
board  sensor  cues.  Making  the  off-boresight  heading  tape  shorter  and  changing  the  indication  to  “helmet 
heading  will  reduce  clutter  and  give  operational  pilots  more  important  tactical  information.  The  BL+ 
off-boresight  heading  display  should  be  modified  to  represent  helmet  heading  and  be  less 
distracting  (R4). 


AIR-TO-GROUND  TASKS 

Compare  the  BL,  BL+,  and  ANDFR  symbology  sets  as  aids  for  off-boresight  air-to-ground 
operational  piloting  tasks. 


Medium  Altitude  Close  Air  Support  (CAS)  Test  Procedures: 

Pilot  workload  and  situation  awareness  ratings  were  measured  while  maintaining  airspeed  and 
altitude  within  preset  bounds  during  close  air  support  (CAS)  target  search.  The  operationally 
representative  task  was  accomplished  in  a  medium  altitude  orbit  while  receiving  targeting  instructions  or 
forward  air  control,  airborne  (FAC(A)).  The  task  was  performed  using  the  BL,  BL+,  and  ANDFR 
symbology  sets. 

The  evaluators  established  a  2  to  6  nautical  mile  (nm)  radius  CAS  orbit  at  18000  ft  MSL  and  400 
KIAS  (Figure  16).  The  SP  assisted  the  evaluator  in  maintaining  desired  orbit  radius  by  verbal 
notification  when  within  1  nm  of  the  radius  limit.  The  evaluators  called  “ready”  after  they  were 
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established  in  the  CAS  orbit.  The  SP  began  an  air  strike  control  talk-on,  started  the  two-minute  event 
clock  which  allowed  VISTA  to  record  the  evaluator’s  off-boresight  time.  The  SP  talked  the  evaluator’s 
eyes  onto  the  target  and  describing  the  target  area  details  to  the  evaluator  for  two  minutes.  A  set  of 
randomized,  standard  targets,  talk-on  techniques,  and  ground  references  were  used  by  the  SP  acting  as  a 
FAC(A).  The  test  points  were  terminated  after  two  minutes  and  called  complete. 

If  the  evaluator’s  altitude  deviated  by  more  than  ±2000  ft  or  the  CAS  wheel  radius  limit  of  2  to  6 
nm  was  breached  before  the  two  minutes  were  up,  the  maneuver  was  terminated  and  repeated.  If  the 
evaluator  successfully  acquired  the  target  before  two  minutes  elapsed,  the  evaluator  continued  to  describe 
target  area  details  to  maintain  an  operationally  representative  workload  until  the  two  minute  limit  was 
reached.  See  Figure  16  for  an  illustration  of  the  CAS  target  search  task. 

The  evaluator  attempted  to  maximize  off-boresight  target  area  search  time  and  minimize  inside- 
the-cockpit  tasks.  The  SP  performed  aircraft  system  crosschecks  and  visual  clearing  during  the  task.  The 
VISTA  recorded  the  amount  of  time  that  the  evaluator’s  helmet  was  30°  or  greater  off-boresight.  VISTA 
displayed  percentage  of  off-boresight  time,  altitude  maintenance,  and  airspeed  maintenance  in  terms  of 
desired,  adequate  or  unsatisfactory  performance.  After  the  task,  the  evaluator  assigned  a  MCH  rating  and 
a  CLSA  rating  using  the  scales  in  Appendix  A. 


The  ratings  were  based  on  the  following  criteria: 

Desired:  Altitude  maintenance:  18000  ft  ±500  ft 

Airspeed  maintenance:  400  KIAS  ±20  KIAS 
Adequate:  Altitude  maintenance:  18000  ft  ±800  ft 

Airspeed  maintenance:  400  KIAS  ±40  KIAS 
Unsatisfactory:  Failure  to  meet  adequate  criteria 

Percent  time  off-boresight  was  also  recorded  for  comparison  and  could  be  measured  as  desired, 
adequate  or  unsatisfactory.  However,  this  was  not  used  to  determine  the  task  performance  rating. 


Off-boresight  visual  time 

Desired:  85%  off-boresight  visual  time  into  the  turn 

Adequate:  65%  off-boresight  visual  time  into  the  tum 
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The  evaluator’s  comments  and  ratings  were  recorded  during  and  after  each  task/test  point.  The 
comments  and  ratings  were  gathered  by  means  of  a  questionnaire  (Appendix  G)  immediately  following 
the  operational  task.  Voice  and  questionnaire  comments  were  compiled  into  daily  flight  reports  for 
results,  conclusions,  and  recommendations.  The  cockpit  voice  recordings,  HUD,  HMD  and  VSS/MFD 
displays  were  also  recorded  and  viewed  post-flight  to  assist  in  writing  flight  reports. 


Medium  Altitude  CAS  Test  Results: 

Pilot  workload  ratings  (MCH),  situation  awareness  ratings  (CLSA)  and  off-boresight  times  using  the 
ANDFR  were  not  all  better  than  the  BL  and  BL+  symbology  sets.  See  Appendix  A  and  Reference  13  for 
more  information  on  the  MCH  ratings  and  CLSA  scales  used  during  testing.  The  results  of  the  EP  tests 
are  shown  in  Figures  17  through  20. 
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Figure  17  is  a  histogram  of  all  MCH  ratings  given  by  the  EPs  after  each  CAS  task.  The  ratings 
were  divided  according  to  HMD  symbology  used,  and  each  histogram  bar  height  represents  the  total 
number  of  tasks  that  received  that  specific  rating.  MCH  ratings  were  assigned  based  on  the  mental 
workload  and  overall  performance  during  each  task.  Lower  MCH  rating  numbers  were  better.  Tasks 
accomplished  successfully  with  acceptable  mental  workload  received  ratings  of  1  through  3.  Tasks 
accomplished  with  an  unacceptably  high  workload  received  ratings  of  4  through  6.  Ratings  of  7  through 
9  described  a  task  that  was  not  accomplished  without  errors  and  with  major  difficulty.  A  task  that  was 
impossible  to  accomplish  received  a  rating  of  10. 

Figure  18  is  a  histogram  of  all  CLSA  ratings  given  by  the  EPs  after  each  CAS  task.  The  ratings 
were  divided  according  to  HMD  symbology  used,  and  each  histogram  bar  height  represents  the  total 
number  of  tasks  that  received  that  specific  rating.  Lower  CLSA  ratings  were  better.  Tasks  accomplished 
with  greater  than  adequate  situation  awareness  (SA)  were  assigned  a  rating  of  1  through  3.  Tasks 
accomplished  with  less  than  adequate  SA  were  assigned  a  rating  of  4  or  5. 

Figure  19  is  a  histogram  of  the  EP’s  average  percent  time  off-boresight  for  each  symbology  set. 
However,  it  was  not  used  to  determine  the  task  performance  rating.  Each  CAS  task  was  assigned  an 
overall  performance  rating  of  desired,  adequate,  or  unsatisfactory,  based  on  airspeed  and  altitude 
maintenance.  Figure  20  is  a  histogram  of  the  average  percent  time  off-boresight  for  each  performance 
level  and  symbology  set. 

MCH  and  CLSA  scales  are  subjective  evaluations  of  pilot  workload,  task  performance  and 
situation  awareness.  Therefore,  objective  statistical  analysis  was  not  used  to  determine  conclusions. 
However,  subjective  conclusions  were  drawn  from  the  data  and  pilot  comments. 
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Figure  17:  Modified  Cooper-Harper  (MCH)  Ratings  for  CAS  Tasks 
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Figure  18:  China  Lake  Situational  Awareness  (CLSA)  Ratings  for  CAS  Tasks 


Figure  19:  Percentage  of  Time  Spent  Looking  Off-Boresight  During  CAS  Tasks 
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Figure  20:  Overall  Performance  versus  Percentage  of  Time  Spent  Looking  Off-Boresight  During 

CAS  Tasks 


The  ANDFR  China  Lake  Situation  Awareness  ratings  (CLS  A)  and  off-boresight  times  were  better 
than  those  from  BL  and  BL+.  However,  the  BL  pilot  workload  ratings  (MCH)  were  much  better  than  the 
other  two  symbology  sets.  The  BL  also  provided  the  best  performance  results  (see  Figure  20).  Using 
ANDFR  allowed  for  more  time  off-boresight  to  be  maintained,  but  with  a  large  reduction  in  task 
performance  over  the  BL.  Therefore,  the  test  team  concluded  the  ANDFR  was  not  better  than  the  BL  and 
BL+  during  the  CAS  tasks. 

The  BL  symbology  was  the  least  distracting  and  allowed  for  an  easy  crosscheck  between 
altitude/airspeed  and  target  search  while  operating  in  a  day  VFR  environment.  The  similarity  of  the 
altitude  and  airspeed  dials  to  the  HUD  display  did  not  require  a  mental  interpretation  adjustment  and 
allowed  pilots  to  concentrate  more  on  finding  the  target.  It  was  also  preferred  due  to  fighter  training 
paradigms  in  which  the  HUD  and  BL/BL+  airspeed/altitude  cues  are  in  the  same  general  location  and 
have  sufficient  analog  trend  information  to  accomplish  mission  tasks.  The  EP  workload  was  lower  with 
the  BL  symbology  set  and  SA  during  task  execution  was  rated  very  good. 

The  shortfalls  of  the  ASAR  were  the  fact  that  the  attitude  reference  was  not  precise  enough  to  fly 
the  aircraft  with  the  precision  required  for  the  task  and  its  location  was  too  high  in  the  pilot’s  field  of 
view  for  this  medium  altitude  air-to-ground  task.  Interpreting  the  ANDFR  digital  odometer-style 
indicators  for  trend  information  caused  an  increase  in  workload. 
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Low  Altitude  Pop  Attack  Test  Procedures: 


Ground  and  flight  tests  were  conducted  by  three  different  EPs  using  the  same  attack  procedure. 
Ground  tests  were  accomplished  using  the  VISTA  VSS  system  to  display  a  simulated  target  with  a 
representative  relative  movement  of  a  ground  target  during  the  attack.  Flight  tests  were  conducted  using 
unfamiliar  targets  and  ground  references  so  EPs  were  required  to  perform  more  target  area  search  during 
the  climb.  The  EPs  attempted  to  maximize  off-boresight  visual  search  and  maintenance  time. 

The  EPs  ingressed  direct  to  the  target  at  500  feet  AGL  and  420  KIAS.  The  SP  armed  the 
VISTA’s  instrumentation  system  timing  prior  to  the  action  point.  At  4.5  nm  from  the  target,  the 
evaluation  pilot  turned  30°  away  from  the  attack  heading  using  a  4  g  level  turn  and  then  rolled  wings 
level  (a  check  -turn).  The  pilot  then  initiated  a  30°  climb  using  4  g  in  2  seconds.  The  climb  continued 
until  roll-in  was  initiated  at  3200  ft  above  the  target  elevation  (“pull  down”  altitude).  The  VISTA 
recorded  the  amount  of  time  that  the  pilot’s  helmet  was  20°  or  greater  off-boresight  towards  the  target. 
Off-boresight  timing  started  after  the  check  turn  rollout  (<19°  bank)  and  continued  until  the  roll-in  after 
the  climb  (>21°  bank).  Average  climb  angle  was  computed  from  4  seconds  after  the  check  turn  rollout 
(<19°  bank)  to  the  roll-in  after  the  climb  (>21°  bank).  The  VISTA  instrumentation  system  monitored  and 
displayed  the  percentage  of  off-boresight  time  and  average  climb  angle  maintenance  in  terms  of  desired 
or  adequate  performance.  Roll-in  altitude  (altitude  when  bank  >21°)  was  also  displayed  by  VISTA. 

After  pull  down,  the  EP  continued  the  attack  with  a  simulated  20°  dive  bomb  delivery  and  subsequent 
climbing  safe  escape  maneuver.  The  safe  escape  maneuver  was  flown  using  5  g  in  2  seconds  until  30° 
nose  high.  The  pilot  performed  a  2  to  4  g  slicing  turn  back  to  low  altitude  while  changing  heading  by  at 
least  90°.  See  Figure  21  for  an  illustration  of  the  pop  attack.  The  minimum  airspeed  allowable  during 
the  maneuver  was  300  KIAS.  The  maximum  allowable  airspeed  was  450  KIAS  due  to  the  Viper  IV 
helmet's  ejection  limit.  The  safety  pilot  performed  aircraft  system  crosscheck,  visual  clearing,  and 
monitored  ground  clearance  altitudes  and  dive  angles.  After  the  task,  the  EP  assigned  a  MCH  rating  and 
CLSA  rating  using  the  scales  in  Appendix  A.  The  MCH  ratings  were  based  on  the  following  criteria: 

Desired:  Climb  angle  maintenance:  Average  30°  ±  2° 

Pull-down  altitude  tolerance:  Planned  ±  200  feet 
Adequate:  Climb  angle  maintenance:  Average  30°  ±  3° 

Pull-down  altitude  tolerance:  Planned  ±  400  feet 
Unsatisfactory:  Failure  to  meet  adequate  criteria 

Percent  time  off-boresight  was  also  recorded  for  comparison  and  could  be  measured  as  desired, 
adequate  or  unsatisfactory.  However,  it  was  not  used  to  determine  the  task  performance  rating. 

Off-boresight  visual  time: 

Desired:  70%  off-boresight  visual  time 

Adequate:  50%  off-boresight  visual  time 

The  EP’s  verbal  comments  and  ratings  were  recorded  during  and  after  each  test  point.  Each  EP 
completed  a  questionnaire  sheet  immediately  following  the  tasks  (ground  tests)  or  sortie  (flight  tests),  and 
completed  a  daily  flight  report  following  the  flight  test  sortie.  The  verbal  comment  tapes  and  the 
questionnaire  comments  were  included  in  the  daily  reports  (Reference  12). 
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Low  Altitude  Pop  Attack  Test  Results: 

The  results  of  the  tests  are  shown  in  Figures  22  through  25.  Figure  22  is  a  histogram  of  all  MCH 
ratings  given  by  the  EPs  after  each  Pop  Attack  task.  The  ratings  are  divided  according  to  HMD 
symbology  used,  and  each  histogram  bar  represents  the  total  number  of  tasks  receiving  that  rating. 
MCHRs  were  assigned  based  on  the  mental  workload  and  overall  performance  during  each  task.  Each 
Pop  Attack  task  was  assigned  an  overall  performance  rating  of  desired,  adequate,  or  unsatisfactory,  based 
on  airspeed  and  altitude  maintenance.  Lower  MCH  numbers  were  better.  Tasks  accomplished 
successfully  with  acceptable  mental  workload  received  ratings  of  1  through  3.  Tasks  accomplished,  but 
only  with  an  unacceptably  high  workload  received  ratings  of  4  through  6.  Ratings  of  7  through  9 
described  a  task  that  was  accomplished  with  errors  and  major  difficulty.  A  task  that  was  impossible  to 
accomplish  reliably  received  a  rating  of  10. 

Figure  23  is  a  histogram  of  all  CLSA  ratings  given  by  the  EPs  after  each  Pop  Attack  task.  Lower 
CLSA  ratings  were  better.  See  Appendix  A  and  Reference  13  for  more  information  on  the  MCH  and 
CLSA  scales  used  during  testing.  Percent  time  off-boresight  was  also  recorded  for  comparison  and  could 


27 


be  measured  as  desired,  adequate  or  unsatisfactory.  However,  it  was  not  used  to  determine  the  task 
performance  rating  (Figure  24).  Figure  25  displays  task  performance  ratings  versus  the  tasks  average 
percent  time  off-boresight  for  each  symbology. 

MCH  and  CLSA  scales  are  subjective  evaluations  of  pilot  workload,  task  performance  and  situation 
awareness.  Therefore,  objective  statistical  analysis  was  not  used  to  determine  conclusions.  However, 
subjective  conclusions  were  drawn  from  the  data  and  pilot  comments. 
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Figure  22:  Modified  Cooper-Harper  (MCH)  Ratings  for  Pop  Attack  Tasks 
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Figure  23:  China  Lake  Situation  Awareness  (CLSA)  Ratings  for  Pop  Attack  Tasks 
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Figure  24:  Percentage  of  Time  Spent  Looking  Off-Boresight  During  Pop  Attack  Tasks 
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Figure  25:  Overall  Performance  versus  Percentage  of  Time  Spent  Looking  Off-Boresight  During 

Pop  Attack  Tasks 


Pilots  had  better  SA  during  the  task  with  BL  than  they  did  with  ANDFR.  In  addition,  task 
performance  was  better  using  the  BL+  symbology  than  with  the  ANDFR.  Pilots  liked  the  consolidated 
display  of  the  ANDFR,  but  the  altitude  and  airspeed  odometers  were  difficult  to  read  under  dynamic 
conditions.  This  made  it  difficult  to  quickly  interpret  the  altitude  and  degraded  the  pilot’s  ability  to 
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anticipate  when  to  role  in  to  the  target.  The  need  for  quick  interpretation  of  altitude  and  climb  angle  was 
the  driving  factor  in  the  task.  The  pilot  should  be  able  to  use  most  all  off-boresight  time  searching  for  the 
target  and  threats.  Transitioning  from  using  the  HUD’s  analog  dials  to  the  ANDFR  odometers  required 
greater  mental  processing  and  significantly  more  time  staring  at  the  display  in  order  to  correctly  interpret 
the  information  and  how  fast  it  was  changing.  This  resulted  in  an  objectionably  small  amount  of  time 
available  to  find  the  target.  It  would  be  easier  to  glance  in  the  HUD  for  the  altitude  and  airspeed 
information  than  to  focus  on  the  odometers.  In  addition,  the  placement  of  the  ANDFR  for  air-to-ground 
tasks  was  too  high.  During  off-boresight  target  search,  pilot’s  eyes  scanned  low  in  the  HMD  field  of 
view  (FOV)  without  moving  their  heads  much  in  elevation.  As  a  result,  the  crosscheck  of  the  HMD 
displayed  symbology  required  the  pilot  to  scan  higher  in  the  HMD  FOV  for  the  information.  The 
ANDFR  was  placed  so  high  that  the  pilot’s  eyes  had  to  scan  up  an  excessive  distance  and  led  to  mental 
strain,  increased  time  to  gather  the  desired  information,  and  distraction  from  the  target.  The  transition 
and  interpretation  difficulties  associated  with  the  ANDFR  would  inevitably  cause  pilots  to  become 
annoyed  with  the  symbology  and  ignore  it  completely.  In  this  case,  the  ANDFR  may  not  be  used  when  it 
was  needed  for  unusual  attitude  recognition. 

During  a  daytime  low  altitude  pop  attack,  pilots  preferred  using  the  BL  symbology.  The 
transition  to  off-boresight  was  easy  and  the  cross-check  scan  pattern  did  not  change  when  retrieving 
altitude  and  airspeed  information.  A  typical  pilot  comment  was  <4The  cross-check  and  interpretation  of 
the  symbology  was  so  intuitive  that  I  could  focus  all  my  mental  effort  on  trying  to  find  the  target.” 
(Reference  12).  Requiring  a  pilot  to  change  his  mental  habit  patterns  when  transitioning  from  on  to  off- 
boresight  should  be  minimized  if  possible.  Pilots  did  need  to  look  inside  one  or  two  times  for  flight  path 
angle  information.  However,  pilots  naturally  retrieved  this  information  very  quickly  by  moving  their 
heads  a  small  amount  while  shifting  their  eyes  across  the  majority  of  the  angular  distance  into  the  HUD. 
The  Baseline  symbology  should  be  used  as  the  default  off-boresight  symbology  set  (R5). 

Pilots  liked  using  the  BL+  symbology  second  best.  The  similarity  of  the  altitude  and  airspeed 
dials  made  transition  to  off-boresight  and  symbology  interpretation  easy.  Although  pilots  did  not  need  an 
attitude  reference  during  a  day  pop  attack,  they  did  like  having  the  attitude  reference  available  for  use  in 
determining  flight  path  angle  in  the  climb.  In  addition,  the  ASAR  was  useful  when  maneuvering  back  to 
low  altitude  after  the  attack  and  looking  back  at  the  target  area  for  threats.  The  ASAR  could  potentially 
aid  in  low  altitude  attacks  at  night  and  potentially  prevent  unusual  attitudes  or  spatial  disorientation  when 
off  target.  However,  the  position  of  the  ASAR  in  the  BL+  symbology  was  not  correctly  programmed 
(too  low  in  the  FOV  at  98  mils  below  HMD  center)  for  the  low  altitude  pop  attack  (i.e.  not  IAW  the  test 
plan  at  65  mils  above  HMD  center).  The  ANDFR  location,  high  in  the  FOV,  was  programmed  correctly 
for  the  low  altitude  pop  attack  task  (Figure  26). 
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Figure  26:  Actual  Pop  Attack  Symbologies  Programmed 


Although  the  ANDFR  was  programmed  and  displayed  correctly,  the  BL+  attitude  reference  was 
incorrectly  displayed  low  in  the  HMD  FOV  as  for  the  air-to-air  task  (Figure  26).  The  impact  of  this 
-  programming  error  was  actually  beneficial.  The  high  positioning  of  the  ANDFR  was  objectionable  as 
previously  noted.  However,  the  low  positioning  of  the  ASAR  in  the  BL+  symbology  was  much  better. 
The  lower  position  of  the  ASAR  allowed  for  a  quicker  cross  check  of  flight  path  angle,  reduced  the 
mental  strain  on  the  pilot,  and  allowed  for  more  time  to  be  spent  looking  for  the  target  than  scanning  up 
to  see  and  interpret  the  symbology.  One  pilot  noted  that  it  was  slightly  too  low  because  it  covered  up  the 
top  portion  of  the  target  area.  The  off-boresight  attitude  reference  display  should  be  positioned  low 
in  the  HMD  field  of  view  for  easy  parameter  cross  check  during  air-to-ground  tasks  (R6). 

The  off-boresight  display  of  aircraft  heading  was  not  used  or  needed  during  the  task,  but  the 
positioning  was  so  high  above  where  the  pilots  eyes  were  looking  (low  in  the  HMD  FOV)  that  the 
symbology  was  not  distracting. 


MILITARY  UTILITY 

Helmet  Fitting  and  Stabilization: 

Correct  vertical  placement  of  the  HMD  display  along  the  pilot’s  Frankfort  plane  or  natural  eye 
line  of  sight  was  very  difficult  with  the  test  helmet.  Correct  helmet  fitting  was  critical  and  had  to  be 
adjusted  for  each  aircrew  member  prior  to  flight.  Many  times  the  center  of  the  HMD  display  would  be 
positioned  just  above  the  HUD  symbology  when  looking  on-boresight.  Prior  to  tasks,  aircrew  would  tilt 
the  helmet  forward  on  their  head  in  order  to  correctly  align  the  HMD  display  with  the  HUD.  The  vertical 
and  lateral  freeplay  in  the  helmet  occasionally  caused  the  symbology  to  move  with  the  helmet’s 
movement.  For  accurate  and  precise  off-boresight  weapons  cueing  tasks,  the  helmet  fitting  would  need 
to  be  comfortable  and  ensure  alignment  with  the  pilot’s  natural  sitting  eye  line  of  sight.  In  addition,  the 
HMD  symbology  should  be  stabilized  relative  to  the  aircraft  body  axis.  The  alignment  and  stabilization 
of  the  HMD  symbology  will  be  crucial  to  effective  implementation  of  the  virtual  HUD  concept  in  any 
fighter  type  aircraft  intending  to  use  the  HMD  symbology  for  accurate  and  precise  weapons  employment. 
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Unusual  Attitude  Recoveries: 


The  ASAR  included  in  the  BL+  and  ANDFR  was  useful  in  determining  roll  direction  and  relative 
pitch  attitudes  during  unusual  attitude  recoveries.  Even  though  the  BL  symbology  did  not  greatly 
enhance  the  pilot  s  ability  to  recognize  unusual  attitudes  while  looking  off-boresight,  pilots  still  preferred 
to  fly  with  BL  for  operational  tasks  during  the  day.  Assuming  that  pilots  would  normally  fly  around  with 
the  BL  set,  an  automatic  addition  of  the  ASAR  display  when  the  ground  collision  warning  system 
activates  or  when  the  aircraft  descends  below  the  programmed  radar  altitude  warning  height  could 
provide  pilots  with  important  time-critical  information  to  ensure  recovery. 

The  addition  of  the  ASAR  could  be  beneficial  at  night,  in  IMC,  or  when  a  discemable  horizon 
does  not  exist.  While  operating  with  NVG’s  often  provides  a  better  horizon  at  night,  certain  conditions 
(i.e.  low  moon  illumination)  could  degrade  the  available  horizon.  Having  the  ASAR  displayed  under 
these  conditions  could  aid  aircrew  from  becoming  disoriented  while  looking  off-boresight.  Having  the 
ASAR  displayed  during  IMC  formation  could  assist  wingmen  from  becoming  spatially  disoriented  while 
flying  on  the  flight  lead’s  wing.  Allowing  pilots  to  select  which  symbology  they  want  to  use  (BL,  BL 
w/ASAR,  BL  w/ASAR  and  helmet  heading,  etc.)  would  enable  pilots  to  determine  which  combination  is 
needed  or  desired.  For  example,  pilots  could  fly  with  BL  during  day  VFR  conditions  and  switch  to  BL 
w/ASAR  when  the  weather  deteriorated.  System  integration  of  off-boresight  HMD  symbology  should 
incorporate  a  pilot  programmable  declutter  option  (R7). 


Air-to-Air: 


When  considering  the  addition  of  weapon/sensor  cues  in  the  HMD,  the  BL  symbology  would  be 
the  most  useful  and  least  distracting  off-boresight  symbology.  Additionally,  the  BL  symbology  will 
allow  pilots  to  quickly  determine  their  “energy”  (altitude  and  airspeed)  and  could  be  beneficial  during 
defensive  fighter  maneuvers. 

The  ASAR  attitude  display  was  not  very  beneficial  during  daytime  AAMD  maneuvers,  unless  the 
pilot  s  eyes  were  looking  behind  the  aircraft’s  wingline.  Looking  behind  the  aircraft  when  pulling  g’s 
and  trying  to  adjust  bank  for  level  flight  was  easier  with  the  ASAR  (i.e.  with  BL+  or  ANDFR)  than  with 
just  the  BL  symbology.  The  digital  flight  path  angle  and  analog  bank  were  the  first  indications  to  the 
pilot  that  the  aircraft  was  climbing  or  descending.  This  could  be  critical  information  for  preventing 
unintentional  descents  and  potential  ground  impact  during  low  altitude  defensive  reactions.  Having  the 
ASAR  automatically  added  to  the  BL  set  when  at  low  altitude  (<5000  ft  AGL)  could  cue  the  pilot  to 
avoid  ground  impact  especially  when  looking  behind  the  aircraft  wingline. 

Air-to-Ground: 


When  cross-checking  HUD  information  during  off-boresight  target  search  tasks,  pilots  naturally 
scanned  their  eyes  to  the  HUD  versus  rotating  their  whole  head  (and  helmet)  to  the  HUD  boresight. 
Implementation  of  a  virtual  HUD  in  the  HMD  will  need  to  allow  pilots  to  use  their  peripheral  vision  as 
well  as  a  glancing  look  at  the  azimuth  limits  of  their  eye  scan.  Otherwise,  pilots  will  need  to  rotate  their 
head  more  towards  aircraft  boresight  than  normal  HUD  operations  which  would  increase  pilot  workload, 
increase  the  time  required  to  cross  check  HUD  information,  and  potentially  decrease  task  performance. 
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FUTURE  TESTING 


Testing  was  limited  to  day  VMC  conditions.  Development  of  the  off-boresight  symbology  should 
continue  and  include  the  recommended  enhancements.  Future  testing  should  focus  on  low  altitude  all 
aspect  missile  defense  maneuvers,  medium/low  altitude  night  employment  tasks  and  formation  flights  on 
the  wing  in  weather.  If  possible,  any  future  testing  should  include  sensor  and  weapons  information 
displayed  in  the  HMD.  The  recommended  enhancements  should  be  tested  with  multiple  sensor  and 
weapons  information  included  in  the  display  as  well  as  during  other  than  day  VMC  condition  (R8). 
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CONCLUSIONS  AND  RECOMMENDATIONS 


This  evaluation  was  conducted  during  day  visual  meteorological  conditions  (VMC)  only. 
However,  conclusions  were  made  considering  the  potential  employment  during  night  or  instrument 
meteorological  conditions  (IMC),  as  well  as  with  weapon  and  multi-sensor  information  included  in  the 
helmet-mounted  display  (HMD). 

Pilots  using  the  Advanced  Non-Distributed  Flight  Reference  (ANDFR)  symbology  in  the  HMD 
were  able  to  accurately  recall  more  data  than  when  using  either  the  Baseline  Plus  (BL+)  or  MJL-STD- 
1787C  Head  Up  Display  (HUD)  symbologies.  Pilots  using  BL+  symbology  were  able  to  recall  more  data 
than  with  the  HUD  symbology.  Pilots  generally  preferred  the  ANDFR  to  BL+  or  the  HUD  for  static 
parameter  recall  because  everything  was  found  in  one  place.  However,  the  BL+  and  ANDFR  bank 
information  was  identified  as  being  somewhat  difficult  to  interpret  at  nose  high  attitudes.  A  better 
horizon  reference  would  improve  the  quick  recognition  of  bank  angle  at  any  flight  path  angle  (FPA). 

During  unusual  attitude  (UA)  recovery  flight  tests,  ANDFR  did  not  provide  quicker  reaction 
times  or  more  correct  inputs  than  BL+.  Pilots  showed  a  preference  for  BL+  over  ANDFR  and  BL 
symbologies  during  UA  recoveries.  This  was  due  to  less  mental  workload  when  transitioning  from  off- 
boresight  to  the  HUD  for  completion  of  the  recovery.  The  Arc  Segmented  Attitude  Reference  (ASAR)  in 
the  BL+  and  ANDFR  symbologies  was  found  to  aid  in  UA  recognition  and  recovery.  However,  quickly 
interpreting  nose  high  attitudes  using  the  ASAR  was  consistently  more  difficult  than  nose  low  attitudes 
and  caused  a  few  incorrect  initial  recovery  inputs.  This  was  primarily  due  to  an  insufficient  off-boresight 
horizon  display. 

During  the  medium  altitude  air-to-air  task,  pilots  concluded  that  the  BL  set  would  be  most 
beneficial  when  considering  the  addition  of  weapons  and  on/off-board  sensor  information  in  the  HMD. 
The  BL  symbology  set  was  the  least  distracting,  allowed  for  a  better  cross  check  between  aircraft  flight 
parameters  and  target  location,  and  was  easy  to  use  when  quickly  transitioning  between  on  and  off- 
boresight.  Task  performance  using  the  BL  was  approximately  the  same  as  when  using  BL+  or  ANDFR. 
However,  pilot  workload  was  reduced  and  situation  awareness  was  higher  when  using  BL.  This  allowed 
pilots  to  spend  more  of  the  off-boresight  time  searching  for  the  target  than  interpreting  the  symbology. 
Pilots  felt  the  addition  of  an  off-boresight  radar  altimeter  display  would  aid  in  task  performance  and 
ground  avoidance  during  low  altitude  tasks. 

For  the  medium  altitude  air-to-ground  close  air  support  (CAS)  and  the  low  altitude  pop  attack 
tasks,  BL  was  preferred  during  daytime  employment.  Using  BL  resulted  in  approximately  the  same 
performance  as  the  other  symbologies,  but  was  the  least  distracting,  easiest  to  interpret,  and  allowed  for 
more  target  search  time.  However,  day  employment  with  a  discemable  horizon  did  not  require  an  off- 
boresight  attitude  reference.  The  BL+  could  be  beneficial  during  target  searches  at  night  or  with  no 
horizon  while  still  not  causing  objectionable  distraction  or  difficulties  when  transitioning  to  and  from  the 
HUD.  However,  the  BL+  aircraft  heading  was  not  needed  off-boresight,  and  the  ASAR  was  too  high  for 
easy  cross  check  during  either  air-to-ground  task.  Pilots  did  not  like  the  ANDFR  altitude  and  airspeed 
odometers  for  use  as  trend  information.  The  odometers  were  very  difficult  to  quickly  adapt  too  when 
attempting  to  gain  trend  or  precision  information.  The  similar  format  and  position  of  altitude  and 
airspeed  in  the  BL  (or  BL+)  set  to  that  of  the  HUD  reduced  mental  workload,  reduced  display  fixation 
and  ultimately  increased  available  target  search  time. 

The  following  recommendations  were  made  to  enhance  the  off-boresight  HMD  symbology  for 
improved  operational  task  performance,  situational  awareness  and  safety. 
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The  BL  symbology  was  most  preferred  for  operational  tasks  due  to  the  limited  clutter  and  distraction. 

The  Baseline  symbology  should  be  used  as  the  default  off-boresight  symbology  set.  (R5,  page  30) 

The  BL  symbology  did  not  provide  a  sufficient  attitude  reference  for  recognition  of  unusual  attitudes 
when  looking  off-boresight.  Allowing  the  pilot  to  add  the  Arc  Segmented  Attitude  Reference  (ASAR)  to 
the  BL  set  for  different  mission  environments  would  aid  in  unusual  attitude  recognition. 

System  integration  of  off-boresight  HMD  symbology  should  incorporate  a  pilot  programmable 
declutter  option.  (R7,  page  32) 

The  off-boresight  horizon  reference  did  not  allow  for  quick  recognition  of  bank  at  nose  high  altitudes. 

The  off-boresight  horizon  reference  should  be  improved.  (Rl,  page  10) 

The  ASAR  analog  FP A  display  had  insufficient  resolution  for  quick  interpretation  of  small  flight  path 
angles.  Interpreting  the  digital  FPA  display  required  too  much  time  and  effort. 

The  off-boresight  analog  flight  path  angle  reference  should  be  improved  for  quicker  interpretation 
of  flight  path  angles  less  than  20°.  (R3,  page  20) 

Radar  altitude  was  required  by  the  phots  when  flying  in  close  proximity  to  the  ground. 

A  radar  altitude  display  should  be  added  to  the  off-boresight  display.  (R2,  page  20) 

The  aircraft  heading  tape  in  the  BL+  symbology  set  was  too  distracting  and  rarely  used.  Helmet  heading 
was  deemed  to  be  more  tactically  beneficial  than  aircraft  heading. 

The  off-boresight  heading  display  should  be  modified  to  represent  helmet  heading  and  be  less 
distracting.  (R4,  page  20) 

The  ASAR  was  located  too  high  (65  mils  above  HMD  center  field  of  view)  for  easy  crosscheck  during 
air-to-ground  tasks. 

The  off-boresight  attitude  reference  display  should  be  positioned  low  in  the  HMD  field  of  view  for 
easy  parameter  cross  check  during  air-to-ground  tasks.  (R6,  page  31) 

The  test  symbologies  would  eventually  incorporate  multiple  sensor/weapon  cues  during  formation,  night 
and  IMC  conditions  (realistic  combat  conditions).  Testing  was  not  accomplished  in  these  areas. 

The  recommended  enhancements  should  be  tested  with  multiple  sensor  and  weapons  information 
included  in  the  display  as  well  as  during  other  than  day  VMC  conditions.  (R8,  page  33) 
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VERY  GOOD -1 


•  FULL  KNOWLEDGE  OF  A/C  ENERGY  STATE  /  TACTICAL 
ENVIRONMENT  /  MISSION 


•  FULL  ABILITY  TO  ANTICIPATE /ACCOMODATE 
TRENDS 


GOOD -2 


ADEQUATE -3 


POOR -4 


FULL  KNOWLEDGE  OF  A/C  ENERGY  STATE  /  TACTICAL 
ENVIRONMENT  /  MISSION 

PARTIAL  ABILITY  TO  ANTICIPATE  /  ACCOMMODATE 
TRENDS 


FULL  KNOWLEDGE  OF  A/C  ENERGY  STATE  /  TACTICAL 

ENVIRONMENT  /  MISSION 

SATURATED  ABILITY  TO  ANTICIPATE  / 

ACCOMMODATE  TRENDS 

SOME  SHEDDING  OF  MINOR  TASKS 


FAIR  KNOWLEDGE  OF  A/C  ENERGY  STATE  /  TACTICAL 
ENVIRONMENT  /  MISSION 
SATURATED  ABILITY  TO  ANTICIPATE  / 
ACCOMMODATE  TRENDS 


•  SHEDDING  OF  ALL  MINOR  TASKS  AS  WELL  AS  MANY 
NOT  ESSENTIAL  TO  FLIGHT  SAFETY  /  MISSION 
EFFECTIVENESS 


VERY  POOR -5 


MINIMAL  KNOWLEDGE  OF  A/C  ENERGY  STATE  / 
TACTICAL  ENVIRONMENT  /  MISSION 
OVERSATURATED  ABILITY  TO  ANTICIPATE  / 
ACCOMMODATE  TRENDS 


•  SHEDDING  OF  ALL  TASKS  NOT  ABSOLUTELY 
ESSENTIAL  TO  FLIGHT  SAFETY  /  MISSION 
EFFECTIVENESS 


Figure  Al:  China  Lake  Situation  Awareness  (CLSA)  Scale 
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DIFFICULTY 

DEMANDS  ON  THE  PILOT 

1 

Very  Easy, 

Highly  Desirable 

Pilot  mental  effort  is  minimal  and 
desired  performance  is  easily  attainable 

Easy, 

Pilot  mental  effort  is  low  and 

Desirable 

desired  performance  is  attainable 

Fair, 

Mild  Difficulty 

Acceptable  pilot  mental  effort  is  required  to 
attain  adequate  system  performance 

Minor  but  annoying 
difficulty 

Moderately  high  pilot  mental  effort  is  required  to 
attain  adequate  system  performance 

Moderately  objectionable 
difficulty 

High  pilot  mental  effort  is  required  to 
attain  adequate  system  performance 

A 

Very  objectionable  but 
tolerable  difficulty 

Maximum  pilot  mental  effort  is  required  to 
attain  adequate  system  performance 

A 

Major  difficulty 

Maximum  pilot  mental  effort  is  required  to 
bring  errors  to  moderate  level 

Major  difficulty 

Maximum  pilot  mental  effort  is  required  to 
avoid  large  or  numerous  errors 

Major  difficulty 

Intense  pilot  mental  effort  is  required  to 
accomplish  task,  but  frequent  or  numerous 
errors  persist 

Impossible 


Instructed  task  cannot  be  accomplished 
reliably 


Pilot  Decisions 


Figure  A2:  Modified  Cooper-Harper  (MCH)  Rating  Scale 
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APPENDIX  B:  Test  Resources  and  VISTA  Simulation  System  (VSS) 
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TEST  RESOURCES 


Simulator; 


The  VISTA  ground  simulation  mode  was  used  to  familiarize  aircrew  (pilots, 
engineers,  and  weapons  system  officer)  with  the  HMD  and  the  various  display 
symbologies.  Aircrew  used  VISTA  in  this  ground  simulator  mode  to  become  familiar 
with  the  maneuvers,  recovery  procedures,  and  the  use  of  the  rating  scales  to  be  used  in 
flight  test.  The  VSS  simulation  mode  was  used  for  a  build-up  approach  prior  to  flight 
and  also  to  gather  ground  test  data. 

Each  aircrew  was  scheduled  for  a  VISTA  HMD  familiarization  and  ground  test 
session.  These  sessions  were  administered  with  the  assistance  of  a  Veridian  test 
engineer.  Each  aircrew  performed  recall  of  own-ship  parameters,  practiced  recovery 
from  unusual  attitudes,  flew  simulated  air-to-ground  and  air-to-air  operational  scenarios, 
and  practiced  giving  ratings  and  comments. 

Test  Aircraft: 


Variable-stability  In-flight  Simulator  Test  Aircraft  (VISTA1 NF-16D. 

The  NF-16D  Variable-stability  In-flight  Simulator  Test  Aircraft  (VISTA,  USAF 
S/N  86-0048)  was  a  modified  F-16D  Block  30,  Peace  Marble  II  (Israeli  version)  aircraft 
with  a  Digital  Flight  Control  System  (DFLCS)  using  Block  40  avionics  and  powered  by 
an  FI  10-GE-100  engine.  All  necessary  controls  were  moved  from  the  front  to  the  aft 
cockpit  to  allow  the  pilot  in  command  or  safety  pilot  (SP)  to  fly  from  the  aft  cockpit.  The 
aft  cockpit  had  conventional  F-16  controls  except  the  servo-driven  throttle,  which 
followed  electrical  commands  of  the  front  cockpit  when  the  VISTA  Simulation  System 
(VSS)  was  engaged.  Primary  VSS  controls,  displays,  and  system  engagement  were 
located  in  the  aft  cockpit.  Front  cockpit  included  the  VSS  control  panel  needed  to  engage 
the  variable  feel  center  stick  or  sidestick,  but  the  VSS  system  could  only  be  engaged  from 
the  aft  cockpit.  Front  cockpit  Multi-Function  Displays  (MFDs)  also  reflected  the  aft 
cockpit  MFDs  and  could  be  used  for  simulation  configuration  controls  if  necessary. 

Other  modifications  to  the  aircraft  included  a  higher  flow  rate  hydraulic  system  with 
increased  capacity  pumps  and  higher  rate  actuators  as  well  as  modifications  to  electrical 
and  avionics  systems  required  to  support  VSS  operations. 

The  EP  flew  the  test  points  from  the  front  cockpit  utilizing  the  side  stick 
controller.  Test  points  were  flown  in  either  the  VSS  (unusual  attitudes)  or  F-16  VISTA 
(all  operational  tasks)  operating  modes.  The  rear  cockpit  SP  set  up  the  VSS  computer 
and  the  HMD  configurations,  performed  routine  F-16  flight  procedures,  and  monitored 
the  safety  of  the  evaluations.  At  any  time,  the  SP  could  disengage  the  VSS  and  take 
control  of  the  aircraft. 

Veridian  Flight  Research  developed  VISTA  software  to  implement  the  test 
maneuvers  described  in  this  test  plan.  The  VISTA  instrumentation  system  was  set  up  to 
aid  in  timing,  measuring,  calculating,  displaying,  and  recording  test  events  as  well  as 
maneuver  performance. 

The  layout  of  major  components  in  the  VISTA  F-16  is  shown  on  Figure  Bl. 
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VISTA’s  Display  System: 

Programmable  Display  System  (PDS): 

The  PDS  equipment  installed  in  the  aircraft  included  these  major  components: 

VISTA  Viper-IV  Helmet  Mounted  Display 

Viper-IV  Helmet  with  associated  electronics  unit  and  other  cockpit  units 
and  cable  assemblies 

Helmet  Tracker  Transmitter  (HTT)  and  associated  electronics  units 

HUD  Electronics  Unit  Switching  Unit  (HEUSU) 

HMD  Camera  Electronics  Unit 

A  sketch  of  the  PDS  equipment  is  given  in  Figure  B2.  This  sketch  provides  a 
general  layout  of  the  PDS  equipment  and  its  interconnection.  The  Viper-IV  HMD  and 
associated  tracking  system  were  installed  in  the  front  cockpit  of  VISTA.  The 
programmable  displays  (both  HUD  and  HMD)  were  the  primary  displays  for  the  EP,  who 
occupied  the  front  cockpit. 
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Figure  B2:  Programmable  Display  System  (PDS)  General  Schematic  Diagram 
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Figure  B3:  HMD  Control  Panel 


The  front  seat  HUD  Pilot  Display  Unit  (PDU)  displayed  either  the  nominal  F-16 
HUD  or  the  programmed  HUD  display  from  the  PDS  (i.e.,  the  PDS  HUD).  The  display 


selection  was  activated  by  the  EEUSU  (Figure  B2).  The  HEUSU  was  controlled  by  a 
switch  located  on  the  HMD  control  panel  in  the  front  cockpit  (Figure  B3).  From  this 
switch,  the  programmable  HUD  (forward  switch  position  -  “PDS”)  could  be  selected 
manually  or  the  selection  could  be  made  by  VSS  computer  control  (center  position  - 
“AUTO”). 

The  rear  cockpit,  occupied  by  the  SP,  was  essentially  unaffected  by  the  PDS  with 
the  notable  exception  that  the  PDS  power  switch  was  on  the  avionics  power  panel  in  the 
rear  cockpit  (on  the  non-essential  avionics  bus)  and  video  switches  had  been  added  so  the 
SP  could,  at  any  time,  view  the  same  displays  as  the  EP  (HUD  or  HMD)  on  either  the 
right  Multi-Function  Display  (MFD)  or  Aft  Seat  HUD  Monitor  (ASHM).  The  EP  could 
also  view  the  nominal  F-16  HUD  on  either  the  ASHM  or  right  MFD  at  any  time,  in 
addition  to  the  PDS  HUD  or  HMD. 

Helmet  Mounted  Display  (HMD) 

Viper-IV  HMD 

The  Viper-IV  HMD  and  associated  head  tracking  system  were  installed  in  the 
front  cockpit  of  VISTA.  The  programmable  displays  (both  HUD  and  HMD)  were  the 
primary  displays  for  the  EP.  The  most  prominent  item  of  the  PDS  was  the  Viper  IV 
HMD.  The  BAE  Systems  Viper  IV  HMD,  shown  in  Figure  B4,  was  a  monocular,  40 
degree  field-of-view  stroke  HMD,  installed  on  a  slightly  modified  HGU-86/P  helmet 
shell.  An  MBU-20/P  oxygen  mask  was  used. 


Figure  B4:  VISTA  Viper-IV  HMD 

Head  position  and  angular  orientation  data  were  obtained  via  the  Honeywell 
Advanced  Metal  Tolerant  Head  Tracking  system  installed  in  the  front  cockpit.  These 
data  were  used  to  generate  the  aircraft-stabilized  virtual  HUD  symbology  and  the  off- 
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boresight  HMD  symbologies.  Due  to  a  malfunctioning  helmet  tracker,  the  virtual  HUD 
could  not  be  used  in  this  evaluation. 


VISTA  Simulation  System  (VSS) 

The  VSS  consisted  of  three  flight  qualified  digital  computers  which  interfaced 
with  F-16  DFLCS,  associated  sensors,  signal  conditioners,  and  displays.  For  in-flight 
simulation  VISTA  used  an  implicit  model  following  technique  where  the  aerodynamic 
model  and  VSS  feedback  gains  were  used  to  model  unaugmented  response 
characteristics.  VSS  computers  also  hosted  flight  control  laws,  which  allowed  VISTA  to 
generate  closed  loop  response  characteristics.  VISTA’s  fully  programmable  variable  feel 
system  could  model  non-linearities  such  as  breakout,  friction,  soft-stops,  hard-stops,  and 
multiple  stick  gradients  as  well  as  adjust  stick  frequency  and  damping.  VISTA  had  the 
capability  to  change  stick  characteristics  and  certain  flight  control  gains  during  the  course 
of  a  flight  using  either  MFDs  or  stored  programs.  However,  only  those  gains  and 
characteristics  which  were  previously  ground  simulated  on  VISTA  and  verified  to 
properly  operate  would  be  tested  in  flight.  VSS  also  included  built-in  test  functions, 
software  safety  trips,  safety  trip  reporting,  engage  and  disengage  logic,  and  VISTA 
Vehicle  Integrity  Monitor  (VIM)  logic.  The  F-16  control  laws  were  used  in  VISTA  for 
all  ground  and  flight  tests.  VISTA  flight  control  modes  were  as  follows: 

1)  DISENGAGED  MODE:  The  Safety  Pilot  (SP)  in  the  aft  cockpit  was  in  control  of 
the  aircraft  and  his  inputs  were  processed  through  the  standard  F-16  digital  flight 
control  laws.  The  VSS  was  out  of  the  loop.  This  mode  was  the  default  on  power  up 
and  could  be  entered  directly  from  any  other  mode. 

2)  F-16  MODE:  The  Evaluation  Pilot  (EP)  in  the  front  cockpit  was  in  control  of  the 
aircraft  and  his  inputs  were  processed  through  the  standard  F-16  digital  flight  control 
laws.  The  F-16  Mode  was  entered  from  the  Disengaged  Mode  and  returned  to  the 
Disengaged  Mode  when  any  of  the  safety  trips  were  activated. 

3)  F-16  EMERGENCY  MODE:  The  EP  in  front  cockpit  was  in  control  of  the  aircraft 
and  his  inputs  were  processed  through  the  standard  F-16  digital  flight  control  laws. 
The  Emergency  Mode  could  be  entered  from  any  mode  but  was  intended  for  use 
only  if  the  SP  was  incapacitated  or  the  aft  cockpit  controls  were  malfunctioning.  If 
the  EP  deselected  the  Emergency  Mode,  the  aircraft  reverted  to  the  F-16  Mode.  If 
the  SP  deselected  the  Emergency  Mode,  the  aircraft  reverted  to  the  Disengaged 
Mode. 

4)  VSS  MODE:  The  EP  in  the  front  cockpit  was  in  control  of  the  aircraft  and  his  inputs 
were  processed  through  VSS  using  the  simulated  control  laws  and  aerodynamics. 
The  VSS  Mode  was  entered  from  the  Disengaged  Mode  and  returned  to  Disengaged 
Mode  when  any  of  the  safety  trips  were  activated. 
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APPENDIX  C:  Ground  Simulation  Information  Recall  Test  Data 
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Table  Cl:  BL  Information  Category  Recall  Data 


Information  Recall  Scores  1 

Aircrew 

Heading 

Altitude 

Bank  Angle 

Flight  Path  Angle 

Pilot  1 

0 

0 

0 

r  2 

_ 

6 

0 

0 

1 

0 

0 

0 

0 

2 

1 

0 

0 

0 

2 

1 

0 

0 

0 

2 

0 

0 

0 

0 

1 

0 

Pilot  2 

0 

2 

2 

0 

2 

0 

0 

2 

0 

0 

2 

0 

0 

2 

0 

0 

1 

0 

1 

r  o 

0 

1 

1 

2 

0 

0 

0 

2 

2 

Pilot  3 

0 

0 

0 

0 

0 

0 

0 

2 

0 

0 

0 

2 

0 

0 

0 

0 

2 

0 

6 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

Table  C2:  BL+  Information  Category  Recall  Data 


Table  C3:  ANDFR  Information  Category  Recall  Data 


Information  Recall  Scores  I 

Aircrew 

Heading 

Altitude 

Bank  Angle 

Flight  Path  Angle 

Pilot  1 

0 

0 

0 

2 

2 

0 

2 

0 

2 

2 

0 

6 

0 

2 

2 

0 

0 

0 

0 

2 

0 

2 

0 

0 

2 

0 

2 

0 

2 

h  2 

Pilot  2 

0 

2 

0 

2 

0 

0 

2 

2 

2 

2 

0 

2 

1 

0 

2 

0 

1 

0 

2 

2 

0 

0 

0 

0 

2 

0 

1 

1 

2 

2 

Pilot  3 

0 

1 

2 

0 

0 

0 

2 

2 

0 

0 

0 

1 

0 

0 

2 

0 

0 

0 

0 

2 

0 

2 

0 

0 

2 

0 

0 

2 

0 

0 
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APPENDIX  D:  Statistical  Computations 
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Statistical  Analysis  for  Unusual  Attitude  Recovery  Response  Time 


A  one-tailed  Student's  T  test  was  performed  with  an  alpha  of  0.05. 

Null  Hypothesis:  The  time  to  the  first  significant  input  for  correct  unusual  attitude  recoveries  with  ANDFR 
was  greater  than  or  equal  to  those  performed  with  BL  or  BL+. 

Alternate  Hypothesis:  The  time  to  the  first  significant  input  for  correct  unusual  attitude  recoveries  with 
ANDFR  was  less  than  those  performed  with  BL  or  BL+. 

The  required  T  score  to  reject  the  null  hypothesis  was  -1.86  or  lower. 


Table  Dl:  Statistics  for  Flight  Unusual  Attitude  Recovery  Response  Time 


BL 

BL+ 

ANDFR 

Pilot  1 

■  2.5 " 

1.56 

;  1.74 

2.68 

1.94 

1.64 

1.88 

1.7 

1.98 

1.94 

1.44 

2.56 

Pilot  2 

mmmr'-m 

f&Z&M 

1.36 

1.18 

1.22 

1.44 

1.6 

1.38 

1.42 

i  w&mm 

1.08 

1.24 

0.68 

1.12 

1.42 

6.36 

0.66 

1.06 

1.61 

1.44 

1.42 

Standard 

Deviation 

0.61 

0.27 

0.66 

T  Scores 

ANDFR  vs  BL 
-0.86 

ANDFR  vs  BL+ 

-0.09 

In  the  case  of  both  BL  and  BL+,  we  fail  to  reject  the  null  hypothesis.  The  test  team  cannot  conclude  with 
95%  confidence  that  in-flight  unusual  attitude  response  times  for  ANDFR  were  lower  than  those  of  BL  or 
BL+. 
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Statistical  Analysis  for  Unusual  Attitude  Recovery  Correctness 


A  one-tailed  Student’s  T  test  was  performed  with  an  alpha  of  0.05. 

Null  Hypothesis:  Unusual  attitude  recoveries  performed  with  ANDFR  were  correct  less  often  or  an  equal 
number  of  times  than  those  using  BL  or  BL-k 

Alternate  Hypothesis:  Unusual  attitude  recoveries  performed  with  ANDFR  were  correct  more  often  than 
those  done  using  BL  or  BL+. 

The  required  T  score  to  reject  the  null  hypothesis  was  1.81  or  greater. 


Table  D2:  Statistics  for  Flight  Unusual  Attitude  Recovery  Correctness 


Correctness  of  Recoveries  | 

BL 

BL+ 

ANDFR 

Correct 

11 

10 

9 

Incorrect 

1 

2 

2 

0.82 

Standard 

Deviation 

0.29 

0.39 

0.40 

T  Scores 

ANDFR  vs  BL 
-0.77 

ANDFR  vs  BL+ 

-0.12 

Note:  When  computing  averages  and  standard  deviations  a  zero 
was  used  for  a  correct  recovery  and  one  for  an  incorrect  recovery. 

In  the  case  of  both  BL  and  BL+,  we  fail  to  reject  the  null  hypothesis.  The  test  team  cannot  conclude  with 
95%  confidence  that  unusual  attitude  recoveries  performed  using  ANDFR  are  correct  more  often  than  those 
performed  using  BL  or  BL-k 
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Statistical  Analysis  for  Off-Boresight  Time  During  the  Air-to-Ground  CAS  Task 


A  one-tailed  Student’s  T  test  was  performed  with  an  alpha  of  0.05. 

Tasks  performed  to  the  desired  or  adequate  level  were  weighted  to  accept  the  entire  off-boresight  time. 
Unsatisfactory  task  performance  resulted  in  a  zero  weight  and  no  off-boresight  time  being  accepted. 

Null  Hypothesis:  The  percentage  of  time  spent  looking  off-boresight  (weighted  for  performance)  for  tasks 
completed  using  ANDFR  was  less  than  or  equal  to  the  percentage  of  time  for  tasks  performed  using  BL  or 


Alternate  Hypothesis:  The  percentage  of  time  spent  looking  off-boresight  (weighted  for  performance)  for 
tasks  completed  using  ANDFR  was  greater  than  the  percentage  of  time  for  tasks  performed  using  BL  or 


The  required  T  score  to  reject  the  null  hypothesis  was  2.02  or  greater. 


Table  D3:  Statistics  for  CAS  Performance 


BL  Off- 
Boresight  Time 

Performance 

Achieved 

BL+  Off- 
Boresight  Time 

Performance 

Achieved 

ANDFR  Off- 
Boresight  Time 

Performance 

Achieved 

65 

D 

75 

D 

88 

A 

85 

D 

80 

D 

79 

D 

95 

D 

100 

U 

100 

D 

100 

D 

100 

A 

100 

D 

100 

D 

85 

A 

95 

U 

85 

D 

D 

95 

A 

90 

D 

Weighted  Average 

Weighted  Average 

Weighted  Average 

88.6  _J] 

71.7 

77 

Standard  Deviation 

Standard  Deviation 

Standard  Deviation 

12.15 

36.15 

'  38.56 

T  Score:  BL  vs  ANDFR 

T  Score:  BL+  vs  NDFR 

j  -0.67 

0.31 

Note.  Off-boresight  times  associated  with  Desired  or  Adequate  performance  were  counted  in  their  entirety. 
Off-boresight  times  associated  with  unsatisfactory  performance  were  counted  as  zero 

In  the  case  of  both  BL  and  BL+,  we  fail  to  reject  the  null  hypothesis.  The  test  team  cannot  conclude  with 
95%  confidence  that  the  percentage  of  time  spent  looking  off-boresight  for  tasks  performed  using  ANDFR 
was  higher  than  for  those  performed  using  BL  or  BL+. 
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Statistical  Analysis  for  Off-Boresight  Time  During  the  Air-to-Ground  Pop  Attack  Task 


A  one-tailed  Student's  T  test  was  performed  with  an  alpha  of  0.05. 

Tasks  performed  to  the  desired  or  adequate  level  were  weighted  to  accept  the  entire  off-boresight  time. 
Unsatisfactory  task  performance  resulted  in  a  zero  weight  and  no  off-boresight  time  being  accepted. 

Null  Hypothesis:  The  percentage  of  time  spent  looking  off-boresight  (weighted  for  performance)  for  tasks 
completed  using  ANDFR  was  less  than  or  equal  to  the  percentage  of  time  for  tasks  performed  using  BL  or 
BL+. 

Alternate  Hypothesis:  The  percentage  of  time  spent  looking  off-boresight  (weighted  for  performance)  for 
tasks  completed  using  ANDFR  was  greater  than  the  percentage  of  time  for  tasks  performed  using  BL  or 
BL+. 

The  required  T  score  to  reject  the  null  hypothesis  was  2.02  or  greater. 

Table  D4:  Statistics  for  Pop  Attack  Performance 


BL  Off- 
Boresight  Time 

Performance 

Achieved 

BL+  Off- 
Boresight 
Time 

Performance 

Achieved 

ANDFR  Off- 
Boresight 
Time 

Performance 

Achieved 

80 

D 

75 

D 

75 

D 

90 

D 

75 

D 

85 

A 

D 

85 

D 

85 

D 

1 

U 

90 

D 

70 

A 

70 

A 

60 

u 

95 

U 

60 

U 

60 

U 

70 

u 

A 

Weighted  Average 

Weighted  Average 

Weighted  Average 

51.67 

57.86 

52.50 

Standard  Deviation 

Standard  Deviation 

Standard  Deviation 

40.70 

39.88 

41.08 

T  Score:  BL  vs  ANDFR 

T  Score:  BL+  vs  NDFR 

0.05 

-6.29 

Note:  Off-boresight  times  associated  with  Desired  or  Adequate  performance  were  counted  in  their  entirety. 
Off-boresight  times  associated  with  unsatisfactory  performance  were  counted  as  zero. 

In  the  case  of  both  BL  and  BL+,  we  fail  to  reject  the  null  hypothesis.  The  test  team  cannot  conclude  with 
95%  confidence  that  the  percentage  of  time  spent  looking  off-boresight  for  tasks  performed  using  ANDFR 
was  higher  than  for  those  performed  using  BL  or  BL+. 
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Statistical  Analysis  for  Off-Boresight  Time  During  the  All-Aspect  Missile  Defense  Task 


A  one-tailed  Student's  T  test  was  performed  with  an  alpha  of  0.05. 

Tasks  performed  to  the  desired  or  adequate  level  were  weighted  to  accept  the  entire  off-boresight  time. 
Unsatisfactory  task  performance  resulted  in  a  zero  weight  and  no  off-boresight  time  being  accepted. 

Null  Hypothesis.  The  percentage  of  time  spent  looking  off-boresight  (weighted  for  performance)  for  tasks 
completed  using  ANDFR  was  less  than  or  equal  to  the  percentage  of  time  for  tasks  performed  using  BL  or 


Alternate  Hypothesis:  The  percentage  of  time  spent  looking  off-boresight  (weighted  for  performance)  for 
tasks  completed  using  ANDFR  was  greater  than  the  percentage  of  time  for  tasks  performed  using  BL  or 
BL+. 

The  required  T  score  to  reject  the  null  hypothesis  was  2.02  or  greater. 


Table  D5:  Statistics  for  A  AMD  Performance 


HUDOff- 
Boresight  Time 

Performance 

Achieved 

VCATS  Off- 
Boresight 
Time 

Performance 

Achieved 

NDFR  Off- 
Boresight 
Time 

Performance 

Achieved 

90 

D 

95 

D 

95 

A 

90 

A 

95 

A 

95 

D 

100 

D 

100 

D 

100 

D 

100 

D 

99 

D 

100 

D 

65 

A 

70 

A 

90 

A 

80 

A 

85 

A 

99 

1  A 

Weighted  Average 

Weighted  Average 

Weighted  Average 

87.5 

^  90.7 

96.5 

Standard  Deviation 

Standard  Deviation 

Standard  Deviation 

13.32 

11.43 

3.94 

T  Score:  BL  vs  ANDFR 

T  Score:  BL+  vs  ANDFR 

5.11 

3.31 

Note:  Off-boresight  times  associated  with  Desired  or  Adequate  performance  where  counted  in  their 
entirety.  Off-boresight  times  associated  with  unsatisfactory  performance  were  counted  as  zero 

In  the  case  of  both  BL  and  BL+,  we  reject  the  null  hypothesis.  The  test  team  concludes  with  95% 
confidence  that  the  percentage  of  time  spent  looking  off-boresight  for  tasks  performed  using  ANDFR  was 
higher  than  for  those  performed  using  BL  or  BL+. 
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APPENDIX  E:  Unusual  Attitude  Test  Data 
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Table  El:  Flight  Test  Unusual  Attitude  Recovery  Data 


Time  to  First  Significant  Input  | 

Task  Number 

Pilot  #1 

Pilot  #2 

Pilot  #3 

BL 

BL+ 

ANDFR 

BL 

BL 

BL+ 

ANDFR 

1 

2.50 

1.56 

1.74 

1.62 

1.24 

1.20 

1.08 

1.24 

0.78 

2 

2.68 

1.94 

— 

1.36 

1.18 

1.22 

0.86 

0.68 

3 

1.88 

1.70 

BUl 

1.44 

1.14 

1.38 

1.12 

0.36 

4 

1.94 

1.44 

2.56 

1.42 

1.60 

- 

0.66 

1.06 

0.74 

Mean  Time  to  1st 
Significant  Correct 
Input  (sec) 

2.25 

1.66 

1.98 

1.27 

m 

1.24 

0.52 

Recovery  Correc 

tness 

Incorrect 

0 

o  n 

0 

0 

1 

0 

i 

1 

2 

Correct 

4 

4 

4 

4 

3 

3 

3 

3 

2 

4 

4 

4 

4 

4 

3 

4 

4 

4 

100% 

100% 

100% 

100% 

75% 

100% 

75% 

75% 

50% 

APPENDIX  F:  Operational  Task  Test  Data 
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Table  FI:  AAMD  Task  Performance  Data 


Pilot  #1 


Symbology 


BL 


BL 


BL+ 


BL+ 


ANDFR 


ANDFR 


CLSA 

Rating 


MCH 

Rating 


2_ 

4 


Altitude 


D 


Airspeed 


D 


Percent  Off- 
Boresight  Time 


90 


90 


95 


95 


95 


95 


BL 


BL 


Pilot  #2 


BL+ 


BL+ 


ANDFR 


ANDFR 


BL 


BL 


Pilot  #3 


BL+ 


BL+ 


ANDFR 


ANDFR 


D 


D 


D 


D 


D 


D 


D 


D 

D 


D 


100 


100 


100 


99 


100 


100 


65 


80 


70 


85 


90 


99 


D  -  Desired  Task  Performance 
A  -  Adequate  Task  Performance 
U  -  Unsatisfactory  Task  Performance 
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Table  F2:  CAS  Task  Performance  Data 


Symbology 

CLSA 

Rating 

MCH 

Rating 

Altitude 

Airspeed 

Percent  Off- 
Boresight  Time 

Pilot  #1 

BL 

3 

3 

D 

D 

65 

BL 

2 

3 

D 

D 

85 

BL 

3 

2 

D 

D 

95  i 

BL+ 

2 

3 

D 

D 

85  I 

BL+ 

2 

2 

D 

D 

90 

ANDFR 

2 

3 

A 

D 

88 

ANDFR 

1 

2 

D 

D 

79 

Pilot  #2 

BL 

2 

3 

D 

D 

100 

BL 

1 

2 

D 

D 

100 

BL+ 

3 

5 

U 

A 

100 

BL+ 

2 

3 

A 

D 

100 

2 

3 

D 

D 

100 

ANDFR 

2 

2 

D 

D 

100 

Pilot  #3 

BL 

2 

2 

D 

D 

85 

BL 

1 

2 

D 

D 

90 

BL+ 

2 

3 

A 

D 

85 

BL+ 

1 

2 

D 

D 

90 

ANDFR 

2 

4 

U 

A 

95 

2 

2 

A 

D 

95 

D  -  Desired  Task  Performance 
A  -  Adequate  Task  Performance 
U  -  Unsatisfactory  Task  Performance 
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D  —  Desired  Task  Performance 
A  -  Adequate  Task  Performance 
U  -  Unsatisfactory  Task  Performance 
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APPENDIX  G:  Questionaires 
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Figure  G1 :  Interpretability  of  Symbology  Questionnaire 

Pilot:. _  Date: _ 

Symbol  set  (circle  one):  BL  /  BL+  /  ANDFR 

1)  How  many  fighter  aircraft  piloting  hours  do  you  have? _ 

2)  How  many  heavy  aircraft  piloting  hours  do  you  have? _ 

3)  How  many  hours  in  a  HUD  equipped  aircraft  do  you  have?. 


Not  Confident 


Somewhat  Confident 


Very  Confident 


Using  the  scale  above,  circle  your  response  to  the  following  questions  in  relation  to  the  symbol  set  you  used  during  the  flight  test: 
Consider  determining  the  given  parameter  in  the  context  of  getting  as  much  data  as  possible. 

1)  How  confident  are  you  that  you  could  use  it  for  determining  heading?  1  2  3  4  5 

2)  How  confident  are  you  that  you  could  use  it  for  determining  airspeed?  1  2  3  4  5 

3)  How  confident  are  you  that  you  could  use  it  for  determining  altitude?  1  2  3  4  5 

4)  How  confident  are  you  that  you  could  use  it  for  determining  bank?  1  2  3  4  5 

5)  How  confident  are  you  that  you  could  use  it  for  determining  pitch?  1  2  3  4  5 

Complete  for  BL+  and  ANDFR  (N/A  for  BL)  ~ 

1  2  3  4  5 

Symbol  set  Both  About  My  instrumentation  (HUD) 

Definitely  Superior  The  Same  Definitely  Superior 


Using  the  scale  above,  circle  your  response  to  the  following  questions  comparing  the  symbol  set  you  used  during  the  flight  test  to  the  HUD: 


1)  Which  allows  better  determination  of  heading? 

2)  Which  allows  better  determination  of  airspeed? 

3)  Which  allows  better  determination  of  altitude? 

4)  Which  allows  better  determination  of  bank? 

5)  Which  allows  better  determination  of  pitch? 

6)  Which  allows  better  determination  of  unusual  attitude  recognition? 

7)  Which  would  you  prefer  in  a  combat  display?  (AA  information) 

8)  Which  would  you  prefer  in  a  combat  display?  (AG  information) 

For  7  and  8  consider  HUD+HMD  or  pure  HUD. _ 

Complete  after  seeing  BL+  and  ANDFR _ 

1  2  3  4 

BL+  Both  About 

Definitely  Superior  The  Same 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


5 

ANDFR 

Definitely  Superior 


Using  the  scale  above,  circle  your  response  to  the  following  questions  comparing  the  HMD  symbol  sets: 

1)  Which  allows  better  determination  of  heading? 

2)  Which  allows  better  determination  of  airspeed? 

3)  Which  allows  better  determination  of  altitude? 

4)  Which  allows  better  determination  of  bank? 

5)  Which  allows  better  determination  of  pitch? 

6)  Which  allows  better  determination  of  unusual  attitude  recognition? 

7)  Which  would  you  prefer  in  a  combat  display?  (AA  information) 

8)  Which  would  you  prefer  in  a  combat  display?  (AG  information) 

For  7  and  8  consider  HUD+HMD  or  pure  HUD. 

Answer  the  following  questions:  ~  ~ ~ 

1)  What  element  of  the  symbol  set  did  you  like  the  least?_ _ 

2)  What  element  of  the  symbol  set  did  you  like  the  most?, _ 

3)  What  improvements  would  you  make  to  the  symbol  set? _ 

4)  Any  additional  comments: 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


Figure  G2:  Unusual  Attitude  Recovery  Task  Questionnaire 


Pilot: _  Date: _ 

Symbol  set  (circle  one):  BL  /  BL+  /  ANDFR 


1  2  3  4  5 

Strongly  Disagree  Indifferent/Don't  Know  Strongly  Agree 

Using  the  scale  above,  circle  your  response  to  the  following  questions  in  relation  to  the  symbol  set  you  used  during  the 
flight  test: _ 

1) 1  was  able  to  rapidly  determine  what  unusual  attitude  that  I  was  in  from  the 
projected  symbology. 

2)  At  first  glance  at  the  symbology,  I  was  able  to  determine  which  control  inputs 
were  required  for  recovery. 

3)  As  I  performed  the  unusual  attitude  recovery,  the  symbology  indications 
were  confusing. 

4)  I  was  confident  of  my  spatial  orientation  before  the  recovery. 

5)  I  was  confident  of  my  spatial  orientation  during  the  recovery. 

6)  I  was  confident  of  my  spatial  orientation  after  the  recovery. 


1)  What  element  of  the  symbol  set  did  you  like  the  least?  Why? 

2)  What  element  of  the  symbol  set  did  you  like  the  most?  Why? 

3)  What  improvements  would  you  make  to  the  symbol  set? 

4)  Any  additional  comments: 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 
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Figure  G3:  Air-to-Air  Operational  Task  Questionnaire 

Pilot: _  Date: _ 

Symbol  set  (circle  one):  BL  /  BL+  /  ANDFR 


12  3  4 

Strongly  Disagree  Indifferent/Don't  Know 

5 

Strongly  Agree 

Using  the  scale  above,  circle  your  response  to  the  following  questions  in  relation  to  the  symbol  set  you  used  during  the 
flight  test: 

1)  The  symbology  was  unobtrusive  in  the  HMD  FOV. 

1 

2 

3 

4 

5 

2)  The  symbology  was  easy  to  interpret. 

1 

2 

3 

4 

5 

3)  I  managed  to  adapt  to  the  symbology  early  in  the  flight. 

1 

2 

3 

4 

5 

4)  I  had  to  look  into  the  cockpit  constantly  to  keep  my  SA. 

1 

2 

3 

4 

5 

5)  I  managed  to  use  the  symbology  effectively  off-boresight. 

1 

2 

3 

4 

5 

6)  I  feel  comfortable  using  the  symbology  while  off-boresight 

1 

2 

3 

4 

5 

7)  I  was  disoriented  at  least  once  using  it  off-boresight. 

1 

2 

3 

4 

5 

8)  The  symbology  was  very  helpful  for  the  off-boresight  task. 

1 

2 

3 

4 

5 

9)  I  had  to  really  force  myself  to  avoid  looking  inside. 

1 

2 

3 

4 

5 

10)  I  feel  confident  of  using  this  symbology  off-boresight. 

1 

2 

3 

4 

5 

11)1  won’t  get  into  an  UA  while  using  this  symbology. 

1 

2 

3 

4 

5 

12)  If  I  get  into  a  UA  with  this.  I’m  confident  of  recovering. 

1 

2 

3 

4 

5 

13)  I  liked  this  symbology  set. 

1 

2 

3 

4 

5 

Answer  the  following  questions: 

1.  What  element  of  the  symbol  set  did  you  like  the  least  if  any?  Why? 

2.  What  element  of  the  symbol  set  did  you  like  the  most  if  any?  Why? 

3.  What  improvements  would  you  make  to  the  symbol  set? 

4.  Any  additional  comments: 
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Figure  G4:  Air-to-Ground  Operational  Task  Questionnaire 


Pilot: _  Date: _ 

Symbol  set  (circle  one):  BL  /  BL+  /  ANDFR 

-  -  3  4  5 

Strongly  Disagree  Indifferent/Don’t  Know  Strongly  Agree 

Using  the  scale  above,  circle  your  response  to  the  following  questions  in  relation  to  the  symbol  set  you  used  during  the 
flight  test: 


1)  The  symbology  was  unobtrusive  in  the  HMD  FOV. 

1  2 

3 

4 

5 

2)  The  symbology  was  easy  to  interpret. 

1 

2 

3 

4 

5 

3)  I  managed  to  adapt  to  the  symbology  early  in  the  flight. 

1  2 

3 

4 

5 

4)  I  had  to  look  into  the  cockpit  constantly  to  keep  my  SA. 

1  2 

3 

4 

5 

5)  I  managed  to  use  the  symbology  effectively  off-boresight. 

1 

2 

3 

4 

5 

6)  I  feel  comfortable  using  the  symbology  while  off-boresight 

1 

2 

3 

4 

5 

7)  I  was  disoriented  at  least  once  using  it  off-boresight. 

i 

2 

3 

4 

5 

8)  The  symbology  was  very  helpful  for  the  off-boresight  task. 

1 

2 

3 

4 

5 

9)  I  had  to  really  force  myself  to  avoid  looking  inside. 

1  2 

3 

4 

5 

10)  I  feel  confident  of  using  this  symbology  off-boresight. 

1 

2 

3 

4 

5 

11)1  won’t  get  into  an  UA  while  using  this  symbology. 

i 

2 

3 

4 

5 

12)  If  I  get  into  a  UA  with  this.  I’m  confident  of  recovering. 

1 

2 

3 

4 

5 

13)  I  liked  this  symbology  set. 

1  2 

3 

4 

5 

Answer  the  following  questions: 

1 .  What  element  of  the  symbol  set  did  you  like  the  least  if  any?  Why? 

2.  What  element  of  the  symbol  set  did  you  like  the  most  if  any?  Why? 

3.  What  improvements  would  you  make  to  the  symbol  set? 

4.  Any  additional  comments: 
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Figure  G5:  Military  Utility 

Pilot/ Aircrew: _ _  Date: 

Symbol  set  (circle  one):  BL  /  BL+  /  ANDFR 


UNUSUAL  ATTITUDES 


12  3  4 

Strongly  Disagree  Indifferent/Don’t  Know 

5 

Strongly  Agree 

Using  the  scale  above,  circle  your  response  to  the  following  questions  in  relation  to  the  symbol  set  you  used  during  the 
flight  test  and  provide  tactical/operational  comments  : 

1)  Does  the  symbology  provide  sufficient  cues  to  the  pilot/aircrew  to 
minimize  reliance  on  cockpit  instrument  gauges  during  recovery? 

1  2  3  4  5 

2)  Could  the  pilot/aircrew  maintain  situational  awareness  of  threat  or 
target  information  while  recovering  from  unusual  attitudes? 

1  2  3  4  5 

AIR-TO-AIR 


1)  What  is  the  impact  on  achieving  the  desired  mission  effect  during  an  all-aspect  missile  defense  maneuver? 

2)  What  additional  considerations  must  be  addressed  for  optimizing  the  symbology  to  successfully 
accomplish  air-to-air  tasks? 

3)  In  what  potential  air-to-air  environments  would  the  addition  of  the  symbology  aid  the  pilot/aircrew  in  mission 
accomplishment? 

4)  In  what  potential  air-to-air  environments  would  the  addition  of  the  symbology  cause  distraction  or  have  negative  mission 
impact? 

5)  What  will  be  the  impact  on  how  the  Air  Force  trains  air-to-air  tactics  if  the  addition  of  the  symbology  is  pursued? 

_I^ZZZZIIZ~~  AIR-TO-GROUND 


1)  What  is  the  impact  of  pilot/aircrew  workload  while  attempting  to  achieve  the  desired  mission  effect  during  air-to-ground 
tasks  (CAS  or  pop  attack)? 

2)  Rate  the  projected  performance  of  the  symbology  in  a  night,  communication  intensive,  live  fire,  formation  CAS  combat 
scenario?  (Circle  one)  Do  you  have  any  recommendations  to  improve  mission  capability  with  the  symbology  in  this 
complex  environment? 

EXCELLENT  GOOD  NEUTRAL  POOR  UNSTAISFACTORY 

3)  How  does  the  symbology  enhance  pilot/aircrew  capability  to  achieve  the  desired  mission  effect  during  low  altitude  pop 
attack  weapons  release  air-to-ground  maneuvers? 

4)  What  additional  considerations  must  be  addressed  for  optimizing  the  symbology  to  successfully  accomplish  air-to-ground 
tasks? 

5)  What  will  be  the  impact  on  how  the  Air  Force  trains  air-to-ground  tactics  if  the  addition  of  the  symbology  is  pursued? 
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APPENDIX  H:  Suggested  Symbology  Enhancements 


Allow  the  pilot  to  program  whether  he  wants  the  Arc  Segmented  Attitude 
Reference  (ASAR)  or  the  heading  bar  displayed  in  addition  to  the  Baseline 
symbology  set. 

Add  a  horizon  reference  line,  similar  to  the  HUD,  that  rotates  and  moves  up  or 
down  relative  to  the  off-boresight  flight  path  circle  (Appendix  H). 

Add  a  horizon  reference  line,  similar  to  the  HUD,  for  comparison  with  the  off 
boresight  flight  path  circle  (Appendix  H). 

When  below  5000  ft  AGL,  add  a  digital  radar  altimeter  preceeded  by  an  “R” 
below  the  existing  BL  analog  altitude  dial. 

Shorten  the  heading  tape,  but  keep  the  10°  tick  marks  and  the  boxed  digit 
heading. 

Test  during  low  altitude  all  aspect  missile  defense  maneuvers,  medium  and  low 
altitude  night  employment  tasks,  formation  flight  on  the  wing  in  instrument 
meteorological  conditions  (IMC)  and  with  additional  sensor/weapon  information 
displayed  in  the  HMD. 


Pilot  comments  indicated  that  the  ASAR  had  insufficient  resolution  for  low  climb  and 
dive  angles.  Also,  in  very  nose-high  attitudes,  the  angle  of  bank  of  the  aircraft  was 
difficult  to  interpret. 

One  potential  solution  to  these  problems,  the  addition  of  a  horizon  line,  was  suggested  by 
the  test  team  and  is  shown  in  Figure  HI.  The  suggested  ASAR  horizon  line  is  similar  to 
the  horizon  line  in  the  HUD. 

As  shown  in  the  Figure,  the  full  travel  of  the  line  is  +10°  to  -10°  of  pitch,  but  this  is  an 
arbitrary  limit  and  should  be  adjusted  as  necessary  based  on  further  investigation.  It  is 
expected  that  a  similar  “mil  per  degree”  relationship  as  the  HUD  uses  would  be  most 
intuitive  to  pilots.  Beyond  these  limits,  the  line  changes  from  solid  to  broken,  indicating 
that  the  range  of  travel  has  been  exceeded,  but  still  providing  pilots  with  horizon  location 
and  orientation  information. 

The  line  rotates  with  the  ASAR  around  the  center  aircraft/digital  flight  path  display. 


Figure  HI:  Suggested  ASAR  Horizon  Enhancement  Option 


This  page  intentionally  left  blank. 
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APPENDIX  I:  List  of  Abbreviations,  Acronyms,  and  Symbols 
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List  of  Abbreviations 


Abbreviation 

A/A 

A/C 

AFFTC 

AFRL 

A/G 

ANDFR 

ASAR 

ASHM 

BL 

BL+ 

CAS 

CLSA 

Deg 

DFLCS 

EF 

EP 

EN 

FAC(A) 

FDL 

FOV 

FPA 

HEUSU 

HMD 

HTT 

HUD 

JON 

JOAP 

kts 

LOS 

MCH 

MFD 

MOP 

NDFR 

Nz 

Obj 

PA 

PDS 

PDU 

SA 

Sec 

SP 

SRB 

TA 

TC 

TIM 

UA 

USAFTPS 

VCATS 

VIM 

VISTA 

VSS 


Definition  Units 

Air-to-air 

aircraft 

Air  Force  Flight  Test  Center 
Air  Force  Research  Laboratory 
Air-to-ground 

Advanced  Non-Distributed  Flight  Reference 

Arc  Segmented  Attitude  Reference 

Aft  Seat  HUD  Monitor 

Baseline 

Baseline  Plus 

Close  Air  Support 

China  Lake  Situational  Awareness 

degrees 

Digital  Flight  Control  System 
Evaluation  Flight  Test  Engineer 
Evaluation  Pilot 

Evaluation  Flight  Test  Weapon  Systems  Officer 
Forward  Air  Control,  Airborne 
Fighter  Data  Link 
Field  of  view 

Flight  Path  Angle  deg 

HUD  Electronics  Unit  Switching  Unit 

Helmet  Mounted  Display 

Helmet  Tracker  Transmitter 

Head  Up  Display 

Job  Order  Number 

Joint  Oil  Analysis  Program 

knots  kts 

Line  of  Sight 

Modified  Cooper-Harper 

Multi-Function  Display 

Measure  of  Performance 

Non-Distributed  Flight  Reference 

Load  Factor  in  Z  axis  G 

Objective 

Pressure  altitude  ft 

Programmable  Display  System 
Pilot  Display  Unit 
Situation  Awareness 

seconds  sec 

Safety  Pilot 

Safety  review  Board 

Target  Aircraft 

Test  Conductor 

Technical  Information  Memorandum 
Unusual  Attitude 

United  States  Air  Force  Test  Pilot  School 
Visually  Coupled  Acquisition  Targeting  System 
Vehicle  Integrity  Monitor 
Variable-stability  In-flight  Simulator  Test  Aircraft 
VISTA  Simulation  System 
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